Configurable platform

ABSTRACT

A fluorescence imaging system for imaging an object, the system includes a white light provider that emits white light, an excitation light provider that emits excitation light in a plurality of excitation wavebands for causing the object to emit fluorescent light, a component that directs the white light and excitation light to the object and collects reflected white light and emitted fluorescent light from the object, a filter that blocks light in the excitation wavebands and transmits at least a portion of the reflected white light and fluorescent light, and an image sensor assembly that receives the transmitted reflected white light and the fluorescent light.

REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application Ser.No. 62/354,611, filed Jun. 24, 2016, titled “Configurable Platform,” andto U.S. Provisional Application Ser. No. 62/287,415, filed Jan. 26,2016, titled “Configurable Platform,” which are hereby incorporated byreference in their entirety.

FIELD OF THE INVENTION

The present disclosure relates generally to medical imaging, and moreparticularly to the acquiring and processing of medical images forvisualizing tissue of a subject.

BACKGROUND OF THE INVENTION

Many forms of intraoperative optical imaging are used in surgicalapplications, and such uses are continuing to expand. One area ofparticular growth involves imaging systems that excite and imagefluorescence emitted by endogenous or exogenously introducedfluorophores. Fluorescence imaging capabilities have consequently beenincorporated into a variety of highly specialized imaging equipmenttailored for particular surgical applications, such as, for example,surgical microscopes, laparoscopy towers, vision systems for surgicalrobots and stand-alone wide field (e.g. laparotomy) imaging systems.However, because hospitals and other healthcare institutions desirefluorescence imaging capabilities for a broad range of surgeries, theymust make a substantial investment in purchasing many specializedimaging devices to serve their varied needs.

Some limitations of intraoperative fluorescence imaging devices that areconfigured for use in specific surgeries have been recognized by others,but previous attempts at generating an adequate solution have fallenshort of the desired outcome. Typically, such attempts consist ofadapting fluorescence imaging devices specifically designed for one typeof surgery for use in another type of surgery (e.g., combining anendoscopic fluorescence system with an exoscope). However, because theoriginal product architecture for such devices was established withoutconsideration of the new surgical application, such attempted adaptationmay result in an unacceptable compromise in performance, functionalityand ergonomics.

Furthermore, many optical imaging devices appear to detect only a singlefluorescence excitation and emission waveband, and consequently arelimited to use with only the particular fluorophores utilizing thatsingle excitation/emission waveband. Current optical imaging devicesthat are capable of detecting multiple fluorescence emission wavebandsappear to require multiple cameras where each camera is dedicated to aparticular emission waveband, and yet an additional camera if real timevisible (white) light imaging functionality is desired. However, thesedevices are too large and cumbersome for use in many surgicalapplications.

Thus, systems and methods that provide fluorescence imaging across abroad range of surgical applications are desirable. Systems and methodsthat provide fluorescence imaging using multiple fluorescenceexcitation/emission wavebands are also desirable.

BRIEF SUMMARY OF THE INVENTION

Described here are variations of fluorescence imaging systems andmethods for imaging an object, where the fluorescence imaging system hasa configurable platform. Generally, one variation of a fluorescenceimaging system may include a white light provider that emits whitelight. The imaging system may include an excitation light provider thatemits excitation light in a plurality of non-overlapping excitationwavebands for causing the object to emit fluorescent light. The imagingsystem may include an interchangeable surgery-specific component thatdirects the white light and excitation light to the object and collectsreflected white light and emitted fluorescent light from the object. Theimaging system may include a filter that blocks substantially all lightin the excitation wavebands and transmits at least a substantial portionof the reflected white light and fluorescent light. The imaging systemmay include an image sensor assembly that receives the transmittedreflected white light and the fluorescent light.

Generally, one variation of a fluorescence imaging system may include awhite light provider that emits white light, an excitation lightprovider that emits excitation light in a plurality of non-overlappingexcitation wavebands for causing the object to emit fluorescent light,an interchangeable surgery-specific component that directs the whitelight and excitation light to the object and collects reflected whitelight and emitted fluorescent light from the object, a filter thatblocks substantially all light in the excitation wavebands and transmitsat least a substantial portion of the reflected white light andfluorescent light, and an image sensor assembly that receives thetransmitted reflected white light and the fluorescent light. In somevariations of the systems described here, at least one of the excitationwavebands may be centered at about 405 nm, about 470-480 nm, about 660nm, about 760-780 nm, about 805 nm, or about 750-810 nm.

In some variations of the systems described here, the excitation lightprovider may include at least three excitation light sources. In some ofthese variations, the excitation light provider may include at leastfour excitation light sources. In some of these variations, theexcitation light provider may include at least five excitation lightsources. In some variations, the excitation light provider may includeat least one solid state light source. In some of these variations, theexcitation light provider may include a laser diode. In some of thesevariations, the excitation light provider may include an LED. In somevariations, the excitation light provider may include a non-solid statelight source. In some variations, at least a portion of the excitationlight provider may be coupled to an optical filter that narrows thespectrum of light emitted from the excitation light provider.

In some variations, the white light provider may include a solid statelight source. In some of these variations, the white light provider mayinclude discrete color solid state light sources. In some of thesevariations, the white light provider may include red, green, and blueLEDs or laser diodes. In some variations, the white light provider mayinclude white LEDs. In some variations, the white light provider mayinclude a non-solid state light source.

In some variations, the filter may have an optical density of at least 4for blocking substantially all light in the excitation wavebands. Insome variations, the filter may transmit at least about 90% of thereflected white light and the fluorescent light. In some variations, thefilter may have a transition region of less than about 10 nm betweensubstantially blocked wavelengths and substantially transmittedwavelengths. In some variations, the filter may be integrated with theimage sensor assembly. In some variations, the filter may be integratedwith the surgery-specific component. In some variations, the filter maybe configured to couple to the image sensor assembly and to thesurgery-specific component.

In some variations, the image sensor assembly may include a single imagesensor. In some of these variations, the image sensor may include acolor image sensor. In some of these variations, the image sensorassembly may include a color filter array coupled to pixels of the colorimage sensor. In some variations, the image sensor may be a monochromeimage sensor. In some variations, the image sensor assembly may includea plurality of image sensors. In some variations, the image sensors maybe coupled to at least one spectral splitter. In some variations, theimage sensor assembly may include a solid state image sensor. In some ofthese variations, the image sensor assembly may include CMOS, CCD, orCID technology which may or may not further includeindium-gallium-arsenide or black silicon material.

In some variations, the surgery-specific component, such as aninterchangeable surgery-specific component, may be configured formicrosurgery. In some variations, the interchangeable surgery-specificcomponent may be configured for laparoscopic or endoscopic surgery. Insome variations, the interchangeable surgery-specific component may beconfigured to provide wide field illumination. In some variations, theinterchangeable surgery-specific component may be configured forstereoscopic laparoscopy. In some variations, the surgery-specificcomponent may be designed for at least two different surgicalapplications.

In some variations, the system may include at least one image processorthat receives image signals from the image sensor assembly and processesthe received image signals to generate images from the received imagesignals. In some variations, the system may include at least onecontroller that controls the system to selectively operate in anon-fluorescence mode, a fluorescence mode, and a combinednon-fluorescence and fluorescence mode. In some of these variations, inthe non-fluorescence mode, the controller may cause the white lightprovider to emit white light and the image processor may generate awhite light image based on image signals associated with the reflectedwhite light from the object. In some of these variations, in thefluorescence mode, the controller may cause the excitation lightprovider to emit excitation light and the image processor may generate afluorescence emission image based on image signals associated with thefluorescent light from the object. In some of these variations, in thecombined non-fluorescence and fluorescence mode, the controller maycause at least a portion of the white light or at least a portion of theexcitation light to be pulsed. In some of these variations, the imageprocessor may separate image signals from the image sensor assembly intoa first set of image signals associated with the reflected white lightand a second set of image signals associated with the fluorescent light,and the image processor may generate a white light image based on thefirst set of image signals and a fluorescence emission image based onthe second set of image signals. In some variations, the system mayinclude a display that displays at least one image generated from imagesignals from the image sensor assembly.

Also described here are variations of fluorescence imaging systems forimaging an object, where the fluorescence imaging system is multiplexed.Generally, one variation of a fluorescence imaging system may include alight source assembly including a white light provider that emits whitelight. The imaging system may include an excitation light provider thatemits excitation light in a plurality of non-overlapping excitationwavebands for causing the object to emit fluorescent light. The imagingsystem may include at least one image sensor that receives reflectedwhite light and emitted fluorescent light from the object. The imagingsystem may include an optical assembly located in the optical pathbetween the object and the image sensor comprising a first optics regionthat projects the reflected white light as a white light image onto theimage sensor, and a second optics region that reduces the image size ofthe fluorescent light, spectrally separates the fluorescent light, andprojects the separated fluorescent light in fluorescent images ontodifferent portions of the image sensor. The imaging system may includean image processor that electronically magnifies the fluorescenceimages.

Generally, one variation of a fluorescence imaging system may include alight source assembly including a white light provider that emits whitelight; an excitation light provider that emits excitation light in aplurality of non-overlapping excitation wavebands for causing the objectto emit fluorescent light; at least one image sensor that receivesreflected white light and emitted fluorescent light from the object; anoptical assembly located in the optical path between the object and theimage sensor comprising a first optics region that projects thereflected white light as a white light image onto the image sensor, anda second optics region that reduces the image size of the fluorescentlight, spectrally separates the fluorescent light, and projects theseparated fluorescent light in fluorescent images onto differentportions of the image sensor; and an image processor that electronicallymagnifies the fluorescence images.

In some variations, at least one of the excitation wavebands may becentered at a wavelength falling substantially outside of the visiblelight spectrum (e.g., between about 450 nm and 650 nm). In some of thesevariations, at least one of the plurality of wavebands may be centeredat about 670 nm, about 770 nm, or about 805 nm. In some of thesevariations, the excitation light provider may comprise a firstexcitation light source emitting excitation light centered at about 670nm, a second excitation light source emitting excitation light centeredat about 770 nm, and a third excitation light source emitting excitationlight centered at about 805 nm. In some variations, at least one of theexcitation wavebands may be centered at about 405 nm, or about 470 nm.

In some variations, the system may include a combining optical assemblycoupled to the light source assembly, wherein the combining opticalassembly combines the emitted white light and excitation light from thelight source assembly into a single optical path. In some of thesevariations, the combining optical assembly may include at least onedichroic mirror. In some of these variations, the combining opticalassembly may comprise optical fibers.

In some variations, the optical assembly may comprise a filter thatblocks substantially all light in the excitation wavebands and transmitsat least a substantial portion of reflected white light and fluorescentlight from the object. In some variations, the optical assembly maycomprise a beam splitter that separates the transmitted light into afirst branch of reflected white light and a second branch of fluorescentlight.

In some variations, the second optics region may comprisedemagnification optics that reduce the image size of the fluorescentlight. In some variations, the second optics region may comprise a beamsplitter that spectrally separates the fluorescent light. In somevariations, the beam splitter may be located in that optical path afterthe demagnification optics. In some variations, the beam splitter mayspectrally separate the fluorescent light in paths corresponding to theexcitation wavebands that generated the fluorescent light. In somevariations, the second optics region may comprise an alignment componentthat makes the spectrally separated fluorescent light and the reflectedwhite light follow the equivalent optical path. In some variations, thebeam splitter may spectrally separate the fluorescent light into fourbranches of fluorescent light that are projected as four fluorescentimages onto quadrants of the image sensor. In some of these variations,the ratio of magnification level of the white light image to themagnification level of each of the fluorescent light images projectedonto the image sensor may be about 2:1. In some of these variations, theimage processor may electronically magnify the fluorescent images by afactor of about 2.

In some variations, the first optics region and the second optics regionmay be different regions in a prism. In some variations, the imageprocessor may spatially co-register the white light image and magnifiedfluorescent images. In some variations, the light source assembly maycomprise at least one solid state light source. In some variations, theimage sensor may have a spatial resolution of at least about 4K.

In some variations, the system may include a display that displays atleast one image generated from image signals from the image sensorassembly.

Generally, one variation of a method for fluorescence imaging of anobject may include emitting white light, emitting excitation light in aplurality of excitation wavebands for causing the object to emitfluorescent light, directing the white light and excitation light to theobject, collecting reflected white light and emitted fluorescent lightfrom the object, blocking light in the excitation wavebands andtransmitting at least a portion of the reflected white light andfluorescent light, and receiving the transmitted reflected white lightand fluorescent light on an image sensor assembly. In some variations,at least one of the excitation wavebands may be centered at about 405nm, about 470-480 nm, about 660 nm, about 760-780 nm, about 805 nm, orabout 750-810 nm.

In some variations, the excitation light may be emitted by an excitationlight provider that comprises at least three excitation light sources.

In some variations, the excitation light may be emitted by an excitationlight provider that comprises at least four excitation light sources. Insome variations, the excitation light may be emitted by an excitationlight provider that comprises at least five excitation light sources. Insome variations, the excitation light may be emitted by an excitationlight provider that comprises at least one solid state light source. Insome variations, the excitation light may be emitted by an excitationlight provider that comprises a laser diode. In some variations, theexcitation light may be emitted by an excitation light provider thatcomprises an LED. In some variations, the excitation light may beemitted by an excitation light provider that comprises a non-solid statelight source. In some variations, the excitation light may be emitted byan excitation light provider in which at least a portion of theexcitation light provider is coupled to an optical filter that narrowsthe spectrum of light emitted from the excitation light provider. Insome variations, the white light may be emitted by a white lightprovider that comprises a solid state light source. In some variations,the white light may be emitted by a white light provider that comprisesdiscrete color solid state light sources.

In some variations, the white light may be emitted by a white lightprovider that comprises red, green, and blue LEDs or laser diodes. Insome variations, the white light may be emitted by a white lightprovider that comprises white LEDs. In some variations, the white lightmay be emitted by a white light provider that comprises a non-solidstate light source.

In some variations, blocking light in the excitation wavebands andtransmitting at least a portion of the reflected white light andfluorescent light may be performed by a filter that has an opticaldensity of at least 4 for blocking substantially all light in theexcitation wavebands. In some variations, blocking light in theexcitation wavebands and transmitting at least a portion of thereflected white light and fluorescent light may be performed by a filterthat transmits at least 90% of the reflected white light and thefluorescent light. In some variations, blocking light in the excitationwavebands and transmitting at least a portion of the reflected whitelight and fluorescent light may be performed by a filter that has atransition region of less than 10 nm between substantially blockedwavelengths and substantially transmitted wavelengths.

In some variations, blocking light in the excitation wavebands andtransmitting at least a portion of the reflected white light andfluorescent light may be performed by a filter that is integrated withthe image sensor assembly. In some variations, blocking light in theexcitation wavebands and transmitting at least a portion of thereflected white light and fluorescent light may be performed by a filterthat is integrated with the interchangeable component.

In some variations, blocking light in the excitation wavebands andtransmitting at least a portion of the reflected white light andfluorescent light may be performed by a filter that is configured tocouple to the image sensor assembly and to the interchangeablecomponent.

In some variations, the image sensor assembly may include a single imagesensor. In some variations, the image sensor may be a color imagesensor. In some variations, the image sensor assembly may comprise acolor filter array coupled to pixels of the color image sensor. In somevariations, the image sensor may be a monochrome image sensor. In somevariations, the image sensor assembly may include a plurality of imagesensors. In some variations, the image sensors may be coupled to atleast one spectral splitter. In some variations, the image sensorassembly may include a solid state image sensor. In some variations, theimage sensor assembly may include CMOS, CCD, or CID technology.

In some variations, the image sensor assembly may includeindium-gallium-arsenide or black silicon material. In some variations,directing the white light and excitation light to the object andcollecting reflected white light and emitted fluorescent light from theobject may be performed by an interchangeable component. In somevariations, the interchangeable component may be configured formicrosurgery. In some variations, the interchangeable component may beconfigured for laparoscopic or endoscopic surgery. In some variations,the interchangeable component may be configured to provide wide fieldillumination. In some variations, the interchangeable component may beconfigured for stereoscopic laparoscopy. In some variations, theinterchangeable component may be configured for robotic surgery. In somevariations, the method may further include receiving image signals fromthe image sensor assembly and processing the received image signals togenerate images from the received image signals.

In some variations, the method may further include selectively operatingin a non-fluorescence mode, a fluorescence mode, or a combinednon-fluorescence and fluorescence mode. In some variations, the methodmay further include while in the non-fluorescence mode, emitting whitelight and the generating a white light image based on image signalsassociated with the reflected white light from the object. In somevariations, the method may further include while in the fluorescencemode, emitting excitation light and generating a fluorescence emissionimage based on image signals associated with the fluorescent light fromthe object. In some variations, the method may further include while inthe combined non-fluorescence and fluorescence mode, pulsing at least aportion of the white light or at least a portion of the excitationlight.

In some variations, the method may further include while in the combinednon-fluorescence and fluorescence mode, temporally multiplexing at leasta portion of the white light and/or at least a portion of the excitationlight. In some variations, the method may further include separatingimage signals from the image sensor assembly into a first set of imagesignals associated with the reflected white light and a second set ofimage signals associated with the fluorescent light, and generating awhite light image based on the first set of image signals and afluorescence emission image based on the second set of image signals. Insome variations, the method may further include displaying at least oneimage generated from image signals from the image sensor assembly. Insome variations, the reflected white light and the fluorescent lightreceived at the image sensor may be temporally multiplexed, spatiallymultiplexed, or both temporally multiplexed and spatially multiplexed.

Generally, one variation of a method for fluorescence imaging of anobject includes emitting white light, emitting excitation light in aplurality of excitation wavebands, causing the object to emitfluorescent light, receiving reflected white light and emittedfluorescent light from the object on at least one image sensor, feedingat least part of the reflected light through an optical assembly locatedin an optical path between the object and the image sensor, wherein: afirst optics region of the optical assembly projects reflected whitelight as a white light image onto the image sensor, and a second opticsregion reduces the image size of the fluorescent light, spectrallyseparates the fluorescent light, and projects the separated fluorescentlight as fluorescence images onto different portions of the imagesensor.

In some variations, at least one of the excitation wavebands may becentered at a wavelength falling outside of the visible light spectrum.In some variations, at least one of the plurality of excitationwavebands may be centered at about 670 nm, about 770 nm, or about 805nm. In some variations, the excitation light may be emitted by anexcitation light provider that comprises a first excitation light sourceemitting excitation light centered at about 670 nm, a second excitationlight source emitting excitation light centered at about 770 nm, and athird excitation light source emitting excitation light centered atabout 805 nm. In some variations, at least one of the excitationwavebands may be centered at about 405 nm, or about 470 nm. In somevariations, the method may further include combining the emitted whitelight and excitation light from the light source assembly into a singleoptical path.

In some variations, the emitted white light and excitation light may becombined by a combining optical assembly that comprises at least onedichroic mirror. In some variations, the emitted white light andexcitation light may be combined by a combining optical assembly thatcomprises optical fibers. In some variations, the optical assembly mayinclude a filter that blocks light in the excitation wavebands andtransmits at least a portion of reflected white light and fluorescentlight from the object. In some variations, the optical assembly mayinclude a beam splitter that separates the transmitted light into afirst branch of reflected white light and a second branch of fluorescentlight.

In some variations, the second optics region may include demagnificationoptics that reduce the image size of the fluorescent light. In somevariations, the second optics region may include a beam splitter thatspectrally separates the fluorescent light. In some variations, the beamsplitter may be located in that optical path after the demagnificationoptics. In some variations, the beam splitter may spectrally separatethe fluorescent light in paths corresponding to the excitation wavebandsthat generated the fluorescent light.

In some variations, the second optics region may include an alignmentcomponent that makes the spectrally separated fluorescent light and thereflected white light follow the same optical path. In some variations,the beam splitter may spectrally separate the fluorescent light intofour branches of fluorescent light that are projected as fourfluorescent images onto quadrants of the image sensor. In somevariations, the ratio of magnification level of the white light image tothe magnification level of each of the fluorescent light imagesprojected onto the image sensor may be about 2:1. In some variations,the method may further include an image processor that electronicallymagnifies the fluorescence images. In some variations, the fluorescentimages may be electronically magnified by a factor of about 2.

In some variations, the method may further include spatiallyco-registering the white light image and magnified fluorescent images.In some variations, the first optics region and the second optics regionmay be different regions in a prism. In some variations, the white lightmay be emitted by a light source assembly that comprises at least onesolid state light source. In some variations, the image sensor may havea spatial resolution of at least about 4K. In some variations, themethod may further include displaying at least one image generated fromimage signals from the image sensor assembly. In some variations, themethod may further include temporally multiplexing at least a portion ofthe white light and/or at least a portion of the excitation light. Insome variations, the method may further include electronicallymagnifying at least some of the fluorescence images.

Generally, one variation of a kit for imaging an object may include anyof the systems described herein or any one of the methods describedherein and a fluorescence imaging agent.

Generally, one variation of a fluorescence imaging agent may includethat for use in any of the systems described herein, any of the methodsdescribed herein or any of the kits described herein. In somevariations, imaging an object may include imaging an object during bloodflow imaging, tissue perfusion imaging, lymphatic imaging, or acombination thereof. In some variations, blood flow imaging, tissueperfusion imaging, and/or lymphatic imaging may include blood flowimaging, tissue perfusion imaging, and/or lymphatic imaging during aninvasive surgical procedure, a minimally invasive surgical procedure, orduring a non-invasive surgical procedure. In some variations, theinvasive surgical procedure may include a cardiac-related surgicalprocedure or a reconstructive surgical procedure. In some variations,the cardiac-related surgical procedure may include a cardiac coronaryartery bypass graft (CABG) procedure. In some variations, the CABGprocedure may be on pump or off pump. In some variations, thenon-invasive surgical procedure may include a wound care procedure. Insome variations, the lymphatic imaging may include identification of alymph node, lymph node drainage, lymphatic mapping, or a combinationthereof. In some variations, the lymphatic imaging may relate to thefemale reproductive system.

In some variations, any of the methods, systems, or kits describedherein may be used for lymphatic imaging. In some variations, any of themethods, systems, or kits described herein may be used for blood flowimaging, tissue perfusion imaging, or a combination thereof.

It will be appreciated that any one or more of the above variations,aspects, features and options, including variations, aspects, featuresand options of the fluorescence imaging systems, methods and kits can becombined.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an illustrative schematic of a fluorescence imaging systemwith a configurable platform.

FIG. 2A is an illustrative depiction of an exemplary illumination modulein a fluorescence imaging system. FIGS. 2B and 2C are illustrativedepictions of exemplary variations of an image acquisition module in afluorescence imaging system. FIGS. 2D and 2E are illustrative depictionsof other variations of exemplary illumination modules in a fluorescenceimaging system.

FIG. 3A is a table summarizing exemplary fluorescenceexcitation/emission wavebands and exemplary fluorophores (imagingagents). FIG. 3B is a plot of absorption and emission spectra forselected fluorophores described in FIG. 3A.

FIG. 4A is an illustrative diagram of the spectrum of excitation lightemitted by an exemplary illumination module. FIG. 4B is an illustrativediagram of the spectrum of light that is blocked by an exemplaryfluorescence excitation light blocking filter.

FIG. 5 is a schematic illustration of a multiplexed fluorescence imagingsystem.

FIG. 6A is an illustrative depiction of one variation of an exemplaryoptical assembly in a fluorescence imaging system. FIG. 6B is anillustrative depiction of another variation of an exemplary opticalassembly in a fluorescence imaging system.

FIG. 7 is an illustrative depiction of another variation of an exemplaryoptical assembly in a fluorescence imaging system.

FIG. 8 is an illustrative depiction of another variation of an exemplaryoptical assembly in a fluorescence imaging system.

FIG. 9 is an illustrative schematic of a multi-band fluorescenceexcitation light blocking filter.

FIG. 10 is a plot of responsivity to different wavebands offered bysilicon-based detectors, SiOnyx black silicon detectors, andindium-gallium-arsenide (InGaAs) detectors.

FIG. 11A is a schematic of a NIR-to-visible light upconverter. FIG. 11Bis a schematic of a NIR-to-visible upconverter array in combination withan image sensor.

FIGS. 12A-12D are illustrative depictions of variations ofbeam-splitting prism and sensor configurations.

FIGS. 13A and 13B are perspective and right-side views of a schematic ofa vertical beam-splitting prism.

FIGS. 14A and 14B are perspective and top views of a schematic of ahorizontal beam-splitting prism.

FIG. 15 is a perspective view of a schematic of a beam-splitting prismassembly.

FIGS. 16A-16C are perspective, top, and right-side views of a schematicof another variation of an exemplary optical assembly in a fluorescenceimaging system.

FIGS. 17A-17C are diagrams of exemplary illumination and imageacquisition timing schemes for a fluorescence imaging system.

DETAILED DESCRIPTION OF THE INVENTION

Reference will now be made in detail to implementations and variationsof the invention, examples of which are illustrated in the accompanyingdrawings. Various fluorescence imaging systems, methods, imaging agents,and kits are described herein. Although at least two variations ofimaging systems, methods (e.g., fluorescence imaging system and methodwith a configurable platform and multiplexed fluorescence imaging systemand method), imaging agents, and kits are described, other variations offluorescence imaging systems, methods, imaging agents, and kits mayinclude aspects of the systems, methods, imaging agents, and kitsdescribed herein combined in any suitable manner having combinations ofall or some of the aspects described.

The various systems and methods may be used for imaging an object. Theobject may, for example, include tissue (e.g., tissue having one or moreendogenous or exogenously-introduced fluorophores), but may additionallyor alternatively include any suitable substance or material to beimaged. In some variations, the systems and methods may employ afluorescence imaging agent such as, for example, indocyanine green (ICG)dye (but other suitable imaging agents may be employed). ICG, whenadministered to the subject, binds with blood proteins and circulateswith the blood in the tissue.

In some variations, the fluorescence imaging agent (e.g., ICG) may beadministered to the subject as a bolus injection, in a suitableconcentration for imaging. In some variations where the method isperformed to assess tissue perfusion, the fluorescence imaging agent maybe administered to the subject by injection into a vein or artery of thesubject such that the dye bolus circulates in the vasculature andtraverses the microvasculature. In some variations in which multiplefluorescence imaging agents are used, such agents may be administeredsimultaneously (e.g., in a single bolus), or sequentially (e.g., inseparate boluses). In some variations, the fluorescence imaging agentmay be administered by a catheter. In some variations, the fluorescenceimaging agent may be administered to the subject less than an hour inadvance of performing the measurements for generating the time series offluorescence images. For example, the fluorescence imaging agent may beadministered to the subject less than 30 minutes in advance of themeasurements. In other variations, the fluorescence imaging agent may beadministered at least 30 seconds in advance of performing themeasurements. In some variations, the fluorescence imaging agent may beadministered contemporaneously with performing the measurements.

In some variations, the fluorescence imaging agent may be administeredin various concentrations to achieve a desired circulating concentrationin the blood. For example, in some variations for tissue perfusionassessment where the fluorescence imaging agent is ICG, the fluorescenceimaging agent may be administered at a concentration of about 2.5 mg/mLto achieve a circulating concentration of about 5 μM to about 10 μM inblood. In some variations, the upper concentration limit for theadministration of the fluorescence imaging agent is the concentration atwhich the fluorescence imaging agent becomes clinically toxic incirculating blood, and the lower concentration limit is the limit forinstruments used to acquire the time series of fluorescence images thatdetect the fluorescence imaging agent circulating in blood. In somevariations, the upper concentration limit for the administration of thefluorescence imaging agent is the concentration at which thefluorescence imaging agent becomes self-quenching. For example, thecirculating concentration of ICG may range from about 2 μM to about 10mM.

Thus, in one aspect, the method may comprise administration of afluorescence imaging agent or other imaging agent to the subject, andgeneration or acquisition of the time series of fluorescence imagesprior to processing the image data. In another aspect, the method mayexclude any step of administering the fluorescence imaging agent orother imaging agent to the subject. For instance, the time series offluorescence images may be based on measurements of autofluorescenceresponse (e.g., native tissue autofluorescence or induced tissueautofluorescence), or measurements of a combination of autofluorescenceand fluorescence arising from a fluorescence imaging agent.

In some variations, a suitable fluorescence imaging agent is an agentwhich can circulate with the blood (e.g., a fluorescence dye which cancirculate with a component of the blood such as lipoproteins or serumplasma in the blood) and which fluoresces when exposed to appropriateexcitation light energy. The fluorescence imaging agent may comprise afluorescence dye, an analogue thereof, a derivative thereof, or acombination of these. A fluorescence dye may include any non-toxicfluorescence dye. In some variations, the fluorescence imaging agentoptimally emits fluorescence in the near-infrared spectrum. In somevariations, the fluorescence imaging agent is or comprises atricarbocyanine dye such as, for example, indocyanine green (ICG). Inother variations, the fluorescence imaging agent is or comprisesfluorescein isothiocyanate, rhodamine, phycoerythrin, phycocyanin,allophycocyanin, o-phthaldehyde, fluorescamine, rose Bengal, trypanblue, fluoro-gold, green fluorescence protein, flavins (e.g.,riboflavin, etc.), methylene blue, porphysomes, cyanine dyes (e.g.,cathepsin-activated Cy5 combined with a targeting ligand, Cy5.5, etc.),IRDye800CW, CLR 1502 combined with a targeting ligand, OTL38 combinedwith a targeting ligand, or a combination thereof, which is excitableusing excitation light wavelengths appropriate to each imaging agent. Insome variations, fluorescence imaging agents with long Stokes shifts(e.g., IR1061, CH1100, etc.) may be used. In some variations, ananalogue or a derivative of the fluorescence imaging agent may be used.For example, a fluorescence dye analogue or a derivative may include afluorescence dye that has been chemically modified, but still retainsits ability to fluoresce when exposed to light energy of an appropriatewavelength. In variations where some or all of the fluorescence isderived from autofluorescence, one or more of the fluorophores givingrise to the autofluorescence may be an endogenous tissue fluorophore(e.g., collagen, elastin, NADH, etc.), 5-aminolevulinic Acid (5-ALA), ora combination thereof.

In some variations, the fluorescence imaging agent may be provided as alyophilized powder, solid, or liquid. The fluorescence imaging agent maybe provided in a vial (e.g., a sterile vial), which may permitreconstitution to a suitable concentration by administering a sterilefluid with a sterile syringe. Reconstitution may be performed using anyappropriate carrier or diluent. For example, the fluorescence imagingagent may be reconstituted with an aqueous diluent immediately beforeadministration. Any diluent or carrier which will maintain thefluorescence imaging agent in solution may be used. As an example, ICGmay be reconstituted with water. In some variations, once thefluorescence imaging agent is reconstituted, it may be mixed withadditional diluents and carriers. In some variations, the fluorescenceimaging agent may be conjugated to another molecule, (e.g., a protein, apeptide, an amino acid, a synthetic polymer, or a sugar) so as toenhance solubility, stability, imaging properties or a combinationthereof. Additional buffering agents may optionally be added includingTris, HCl, NaOH, phosphate buffer, HEPES.

A person of skill in the art will appreciate that, although exemplaryfluorescence imaging agents were described above in detail, otherimaging agents may be used in connection with the systems, methods,techniques and kits described herein, depending on the optical imagingmodality.

In some variations, the fluorescence imaging agent used in combinationwith the methods, systems and kits described herein may be used forblood flow imaging, tissue perfusion imaging, lymphatic imaging, or acombination thereof, which may performed during an invasive surgicalprocedure, a minimally invasive surgical procedure, a non-invasivesurgical procedure, or a combination thereof. Examples of invasivesurgical procedure which may involve blood flow and tissue perfusioninclude a cardiac-related surgical procedure (e.g., CABG on pump or offpump) or a reconstructive surgical procedure. An example of anon-invasive or minimally invasive procedure includes wound (e.g.,chronic wound such as for example pressure ulcers) treatment and/ormanagement. In this regard, for example, a change in the wound overtime, such as a change in wound dimensions (e.g., diameter, area), or achange in tissue perfusion in the wound and/or around the peri-wound,may be tracked over time with the application of the methods andsystems. Examples of lymphatic imaging include identification of one ormore lymph nodes, lymph node drainage, lymphatic mapping, or acombination thereof. In some variations such lymphatic imaging mayrelate to the female reproductive system (e.g., uterus, cervix, vulva).

In variations relating to cardiac applications, the imaging agent(s)(e.g., ICG alone or in combination with another imaging agent) may beinjected intravenously through, for example, the central venous line,bypass pump and/or cardioplegia line to flow and/or perfuse the coronaryvasculature, microvasculature and/or grafts. ICG may be administered asa dilute ICG/blood/saline solution down the grafted vessel such that thefinal concentration of ICG in the coronary artery is approximately thesame or lower as would result from injection of about 2.5 mg (i.e., 1 mlof 2.5 mg/ml) into the central line or the bypass pump. The ICG may beprepared by dissolving, for example, 25 mg of the solid in 10 ml sterileaqueous solvent, which may be provided with the ICG by the manufacturer.One milliliter of the ICG solution may be mixed with 500 ml of sterilesaline (e.g., by injecting 1 ml of ICG into a 500 ml bag of saline).Thirty milliliters of the dilute ICG/saline solution may be added to 10ml of the subject's blood, which may be obtained in an aseptic mannerfrom the central arterial line or the bypass pump. ICG in blood binds toplasma proteins and facilitates preventing leakage out of the bloodvessels. Mixing of ICG with blood may be performed using standardsterile techniques within the sterile surgical field. Ten milliliters ofthe ICG/saline/blood mixture may be administered for each graft. Ratherthan administering ICG by injection through the wall of the graft usinga needle, ICG may be administered by means of a syringe attached to the(open) proximal end of the graft. When the graft is harvested surgeonsroutinely attach an adaptor to the proximal end of the graft so thatthey can attach a saline filled syringe, seal off the distal end of thegraft and inject saline down the graft, pressurizing the graft and thusassessing the integrity of the conduit (with respect to leaks, sidebranches etc.) prior to performing the first anastomosis.

Lymphatic mapping is an important part of effective surgical staging forcancers that spread through the lymphatic system (e.g., breast, gastric,gynecological cancers). Excision of multiple nodes from a particularnode basin can lead to serious complications, including acute or chroniclymphedema, paresthesia, and/or seroma formation, when in fact, if thesentinel node is negative for metastasis, the surrounding nodes willmost likely also be negative. Identification of the tumor draining lymphnodes (LN) has become an important step for staging cancers that spreadthrough the lymphatic system in breast cancer surgery, for example. LNmapping involves the use of dyes and/or radiotracers to identify the LNseither for biopsy or resection and subsequent pathological assessmentfor metastasis. The goal of lymphadenectomy at the time of surgicalstaging is to identify and remove the LNs that are at high risk forlocal spread of the cancer. Sentinel lymph node (SLN) mapping hasemerged as an effective surgical strategy in the treatment of breastcancer. It is generally based on the concept that metastasis (spread ofcancer to the axillary LNs), if present, should be located in the SLN,which is defined in the art as the first LN or group of nodes to whichcancer cells are most likely to spread from a primary tumor. If the SLNis negative for metastasis, then the surrounding secondary and tertiaryLN should also be negative. The primary benefit of SLN mapping is toreduce the number of subjects who receive traditional partial orcomplete lymphadenectomy and thus reduce the number of subjects whosuffer from the associated morbidities such as lymphedema andlymphocysts.

The current standard of care for SLN mapping involves injection of atracer that identifies the lymphatic drainage pathway from the primarytumor. The tracers used may be radioisotopes (e.g. Technetium-99 orTc-99m) for intraoperative localization with a gamma probe. Theradioactive tracer technique (known as scintigraphy) is limited tohospitals with access to radioisotopes, requires involvement of anuclear physician, and does not provide real-time visual guidance. Acolored dye, isosulfan blue, has also been used, however this dye cannotbe seen through skin and fatty tissue. In addition, blue stainingresults in tattooing of the breast lasting several months, skin necrosiscan occur with subdermal injections, and allergic reactions with rareanaphylaxis have also been reported. Severe anaphylactic reactions haveoccurred after injection of isosulfan blue (approximately 2% ofpatients). Manifestations include respiratory distress, shock,angioedema, urticaria and pruritus. Reactions are more likely to occurin subjects with a history of bronchial asthma, or subjects withallergies or drug reactions to triphenylmethane dyes. Isosulfan blue isknown to interfere with measurements of oxygen saturation by pulseoximetry and methemoglobin by gas analyzer. The use of isosulfan bluemay result in transient or long-term (tattooing) blue coloration.

In contrast, fluorescence imaging in accordance with the variousembodiments for use in SLN visualization, mapping, facilitates directreal-time visual identification of a LN and/or the afferent lymphaticchannel intraoperatively, facilitates high-resolution optical guidancein real-time through skin and fatty tissue, and facilitatesvisualization of blood flow, tissue perfusion or a combination thereof.

In some variations, visualization, classification or both of lymph nodesduring fluorescence imaging may be based on imaging of one or moreimaging agents, which may be further based on visualization and/orclassification with a gamma probe (e.g., Technetium Tc-99m is a clear,colorless aqueous solution and is typically injected into theperiareolar area as per standard care), another conventionally usedcolored imaging agent (isosulfan blue), and/or other assessment such as,for example, histology. The breast of a subject may be injected, forexample, twice with about 1% isosulfan blue (for comparison purposes)and twice with an ICG solution having a concentration of about 2.5mg/ml. The injection of isosulfan blue may precede the injection of ICGor vice versa. For example, using a TB syringe and a 30 G needle, thesubject under anesthesia may be injected with 0.4 ml (0.2 ml at eachsite) of isosulfan blue in the periareolar area of the breast. For theright breast, the subject may be injected at 12 and 9 o'clock positionsand for the left breast at 12 and 3 o'clock positions. The total dose ofintradermal injection of isosulfan blue into each breast may be about4.0 mg (0.4 ml of 1% solution: 10 mg/ml). In another exemplaryvariation, the subject may receive an ICG injection first followed byisosulfan blue (for comparison). One 25 mg vial of ICG may bereconstituted with 10 ml sterile water for injection to yield a 2.5mg/ml solution immediately prior to ICG administration. Using a TBsyringe and a 30 needle, for example, the subject may be injected withabout 0.1 ml of ICG (0.05 ml at each site) in the periareolar area ofthe breast (for the right breast, the injection may be performed at 12and 9 o'clock positions and for the left breast at 12 and 3 o'clockpositions). The total dose of intradermal injection of ICG into eachbreast may be about 0.25 mg (0.1 ml of 2.5 mg/ml solution) per breast.ICG may be injected, for example, at a rate of 5 to 10 seconds perinjection. When ICG is injected intradermally, the protein bindingproperties of ICG cause it to be rapidly taken up by the lymph and movedthrough the conducting vessels to the LN. In some variations, the ICGmay be provided in the form of a sterile lyophilized powder containing25 mg ICG with no more than 5% sodium iodide. The ICG may be packagedwith aqueous solvent consisting of sterile water for injection, which isused to reconstitute the ICG. In some variations the ICG dose (mg) inbreast cancer sentinel lymphatic mapping may range from about 0.5 mg toabout 10 mg depending on the route of administration. In somevariations, the ICG does may be about 0.6 mg to about 0.75 mg, about0.75 mg to about 5 mg, about 5 mg to about 10 mg. The route ofadministration may be for example subdermal, intradermal (e.g., into theperiareolar region), subareolar, skin overlaying the tumor, intradermalin the areola closest to tumor, subdermal into areola, intradermal abovethe tumor, periareolar over the whole breast, or a combination thereof.The NIR fluorescent positive LNs (e.g., using ICG) may be represented asa black and white NIR fluorescence image(s) for example and/or as a fullor partial color (white light) image, full or partial desaturated whitelight image, an enhanced colored image, an overlay (e.g., fluorescencewith any other image), a composite image (e.g., fluorescenceincorporated into another image) which may have various colors, variouslevels of desaturation or various ranges of a color tohighlight/visualize certain features of interest. Processing of theimages may be further performed for further visualization and/or otheranalysis (e.g., quantification). The lymph nodes and lymphatic vesselsmay be visualized (e.g., intraoperatively, in real time) usingfluorescence imaging systems and methods according to the variousembodiments for ICG and SLNs alone or in combination with a gamma probe(Tc-99m) according to American Society of Breast Surgeons (ASBrS)practice guidelines for SLN biopsy in breast cancer patients.Fluorescence imaging for LNs may begin from the site of injection bytracing the lymphatic channels leading to the LNs in the axilla. Oncethe visual images of LNs are identified, LN mapping and identificationof LNs may be done through incised skin, LN mapping may be performeduntil ICG visualized nodes are identified. For comparison, mapping withisosulfan blue may be performed until ‘blue’ nodes are identified. LNsidentified with ICG alone or in combination with another imagingtechnique (e.g., isosulfan blue, and/or Tc-99m) may be labeled to beexcised. Subjects of the above methods may have various stages of breastcancer (e.g., IA, IB, IIA).

In some variations, such as for example, in gynecological cancers (e.g.,uterine, endometrial, vulvar and cervical malignancies), ICG may beadministered interstitially for the visualization of lymph nodes,lymphatic channels, or a combination thereof. When injectedinterstitially, the protein binding properties of ICG cause it to berapidly taken up by the lymph and moved through the conducting vesselsto the SLN. ICG may be provided for injection in the form of a sterilelyophilized powder containing 25 mg ICG (e.g., 25 mg/vial) with no morethan 5.0% sodium iodide. ICG may be then reconstituted with commerciallyavailable water (sterile) for injection prior to use. According to anembodiment, a vial containing 25 mg ICG may be reconstituted in 20 ml ofwater for injection, resulting in a 1.25 mg/ml solution. A total of 4 mlof this 1.25 mg/ml solution is to be injected into a subject (4×1 mlinjections) for a total dose of ICG of 5 mg per subject. The cervix mayalso be injected four (4) times with a 1 ml solution of 1% isosulfanblue 10 mg/ml (for comparison purposes) for a total dose of 40 mg. Theinjection may be performed while the subject is under anesthesia in theoperating room. In some variations the ICG dose (mg) in gynecologicalcancer sentinel lymph node detection and/or mapping may range from about0.1 mg to about 5 mg depending on the route of administration. In somevariations, the ICG dose may be about 0.1 mg to about 0.75 mg, about0.75 mg to about 1.5 mg, about 1.5 mg to about 2.5 mg, or about 2.5 mgto about 5 mg. The route of administration may be for example cervicalinjection, vulva peritumoral injection, hysteroscopic endometrialinjection, or a combination thereof. In order to minimize the spillageof isosulfan blue or ICG interfering with the mapping procedure when LNsare to be excised, mapping may be performed on a hemi-pelvis, andmapping with both isosulfan blue and ICG may be performed prior to theexcision of any LNs. LN mapping for Clinical Stage I endometrial cancermay be performed according to the NCCN Guidelines for Uterine Neoplasms,SLN Algorithm for Surgical Staging of Endometrial Cancer; and SLNmapping for Clinical Stage I cervical cancer may be performed accordingto the NCCN Guidelines for Cervical Neoplasms, Surgical/SLN MappingAlgorithm for Early-Stage Cervical Cancer. Identification of LNs maythus be based on ICG fluorescence imaging alone or in combination orco-administration with a colorimetric dye (isosulfan blue) and/orradiotracer.

Visualization of lymph nodes may be qualitative and/or quantitative.Such visualization may comprise, for example, lymph node detection,detection rate, anatomic distribution of lymph nodes. Visualization oflymph nodes according to the various embodiments may be used alone or incombination with other variables (e.g., vital signs, height, weight,demographics, surgical predictive factors, relevant medical history andunderlying conditions, histological visualization and/or assessment,Tc-99m visualization and/or assessment, concomitant medications).Follow-up visits may occur on the date of discharge, and subsequentdates (e.g., one month).

Lymph fluid comprises high levels of protein, thus ICG can bind toendogenous proteins when entering the lymphatic system. Fluorescenceimaging (e.g., ICG imaging) for lymphatic mapping when used inaccordance with the methods and systems described herein offers thefollowing example advantages: high-signal to background ratio (or tumorto background ratio) as NIR does not generate significantautofluorescence, real-time visualization feature for lymphatic mapping,tissue definition (i.e., structural visualization), rapid excretion andelimination after entering the vascular system, and avoidance ofnon-ionizing radiation. Furthermore, NIR imaging has superior tissuepenetration (approximately 5 to 10 millimeters of tissue) to that ofvisible light (1 to 3 mm of tissue). The use of ICG for example alsofacilitates visualization through the peritoneum overlying thepara-aortic nodes. Although tissue fluorescence can be observed with NIRlight for extended periods, it cannot be seen with visible light andconsequently does not impact pathologic evaluation or processing of theLN. Also, florescence is easier to detect intra-operatively than bluestaining (isosulfan blue) of lymph nodes.

Fluorescence Imaging System with a Configurable Platform

A fluorescence imaging system may be built upon a platform that can beoperator-configured for use in a variety of surgical applications (e.g.,microsurgery, open field/laparotomy, minimally invasive surgery(laparoscopy/arthroscopy), robotic surgery applications, scintigraphy,etc., or a combination thereof) and that can simultaneously beoperator-configured for use with fluorophores (imaging agents) havingfluorescence excitation/emission wavebands from the UV through thevisible and NIR spectrums, or a selected subset of this range. Previousintraoperative fluorescence imaging devices appear to have historicallybeen conceived and developed with a specific surgical application inmind, and even devices that attempt to add some degree ofuser-configurability are either limited to one or two surgicalconfigurations or to one or two fluorescence wavebands.

In some variations, as shown in FIG. 1, the fluorescence imaging system100 may be structured as an assembly of modular components including oneor more operative modules 110 and one or more modules 120. The modules120 may be surgery-specific (e.g., for a single type of surgery and/orallow selection for a different type of surgery), and may further bereferred to as surgery-specific modules 120. In some variations, thesystem 100 may include one or more accessory modules 130 and/or datamodules 140. Operative modules 110 may include components that providewhite light for illumination and/or light for fluorescence excitation,and components that generate images from reflected white light and/orfluorescence emission light. Surgery-specific modules 120 may couple toone or more of the operative modules in order to establish imagingdevice configurations that are designated for specific types ofsurgeries. Accessory modules 130 may be interconnected with some or allof the other modules and may provide mechanical support or enclosure forone or more of the modules, aid in the interconnection/adaptation ofother modules, and/or perform suitable functions not provided by theother modules. Data modules 140 may be interconnected with some or allof the other modules and provide additional functions such as enablingimage and data display, recording, and/or printing. Although thecomponents of the system are primarily described herein as grouped inthese modules, in some variations, the various components may beorganized and grouped in any suitable manner (that is, the variouscomponents described herein may be combined and arranged in assembliesand subassemblies different from the modules described herein).

In some variations, a fluorescence imaging system for imaging an objectmay include: a white light provider that emits white light, anexcitation light provider that emits excitation light in a plurality ofexcitation wavebands for causing the object to emit fluorescent light,an interchangeable surgery-specific component that directs the whitelight and excitation light to the object and collects reflected whitelight and emitted fluorescent light from the object, a filter thatblocks substantially all light in the excitation wavebands and transmitsat least a substantial portion of the reflected white light andfluorescent light, and an image sensor that receives the transmittedreflected white light and the fluorescent light. In some variations, theexcitation light provider emits non-overlapping excitation wavebands forcausing the object to emit fluorescent light. As used herein,non-overlapping wavebands include substantially non-overlappingwavebands whereby the signal strength of any overlapping portion isminimal relative to the center frequency signal strength. For example,in some variations, the signal strength of any overlapping portion is atleast one order of magnitude less, at least two orders of magnitudeless, at least four orders of magnitude less, or at least 10 orders ofmagnitude or less than the center frequency signal strength.

Operative Modules

In some variations, the operative modules of the fluorescence imagingsystem 100 may include an illumination module 210, an optical imageacquisition module 220, a controller module (and/or a 3D controllermodule), a processor module (and/or a 3D processor module), and/or apost processor/data manager module.

Illumination Module

As shown in FIG. 2A, the illumination module 210 may contain a whitelight provider 212 (with one or more light sources 212 a, 212 b, and 212c) that emits visible (white) light, an excitation light provider 214(with one or more excitation light sources 214 a, 214 b, 214 c, 214 d,and 214 e) that emits excitation light, and optics 211 for manipulatingthe white light and/or excitation light.

The white light provider 212 may include multiple discrete color lightsources (e.g., 212 a, 212 b, and 212 c) that in combination providewhite light, or may include one or more white light sources.Additionally, the white light provider may include light sources thatare solid state (e.g., LEDs, laser diodes, etc.) and/or non-solid state.For example, in one variation, the white light provider may include acombination of discrete color solid state sources such as red, green,and blue LEDs and/or diode lasers. In another variation, the white lightprovider may include white LEDs. In yet another variation, the whitelight provider may include one or more broad spectrum non-solid statesources such as arc lamps, which in some variations may be combined withcolor correction filters. In another variation, the white light providermay include any suitable combination of the above.

The excitation light provider may include one or more light sources(e.g., 214 a-214 d) that emit light in multiple wavebands forfluorescence excitation. The multiple wavebands are preferablynon-overlapping or sufficiently separated from each other such that asingle multi-band fluorescence excitation blocking filter can be used,as described further below. As a result, the need for moving multipleblocking filters into and out of the imaging path may be eliminated insome variations. In some variations, the excitation light provider mayemit fluorescent light in a plurality of non-overlapping excitationwavebands within the ultraviolet (UV), visible, and near-infrared (NIR)spectrum. Each excitation waveband is designated to excite acorresponding endogenous or exogenously-introduced fluorophore, and toresult in a corresponding approximate emission waveband of fluorescentlight emitted from the fluorophore. In an exemplary embodiment, theexcitation light provider may emit light in three or more of theexcitation wavebands shown in FIG. 3A. For example, as shown in FIG. 4A,the excitation light provider may emit light in Band 1 (about 405 nmexcitation light), Band 2 (about 470-480 nm excitation light), Band 3(about 660 nm excitation light), Band 4 (about 760-780 nm excitationlight), and Band 5 or 6 (about 805 nm excitation light). The excitationlight provider may additionally or alternatively emit light in Band 7(about 750-810 nm excitation light). However, the excitation lightprovider may emit light in any suitable number (e.g., 2, 3, 4, 5, 6 orall 7) of Bands 1, 2, 3, 4, 5, 6 and 7 summarized in FIG. 3A, and in anysuitable combination. The absorption and fluorescent emission spectra ofselected exemplary fluorophores from FIG. 3A are illustrated in FIG. 3B.Furthermore, the excitation light provider may additionally oralternatively include one or more light sources that emit light in othersuitable, sufficiently separated wavebands other than those summarizedin FIG. 3A.

As shown in FIG. 2A, the excitation light provider may include multiplelight sources (e.g., 214 a, 214 b, 214 c, 215 d, etc.) where each lightsource is configured to emit light in a defined excitation waveband.Additionally, the excitation light provider may include light sourcesthat are solid state (e.g., LEDs, laser diodes, etc.) and/or non-solidstate. In one variation, the excitation light provider may includenarrow spectrum, solid state sources, such as laser diodes. In anothervariation, the excitation light provider may include broader spectrumsolid state sources, such as LEDs. In yet another variation, theexcitation light provider may include non-solid state sources, such asarc lamps. Broad spectrum light sources (solid state or non-solid state)may be coupled with output spectrum narrowing optical filters that limitand determine the spectrum of light emitted from the lightsource/optical filter subassembly.

As shown in FIG. 2A, the illumination module may include optics 211and/or 216, which may include light combining and projection optics(e.g., lenses, mirrors, dichroics, fiber optics, etc.). In somevariations, these optics may combine the light emitted by the whitelight provider and excitation light provider into a single optical paththat enables light from the multiple light sources to be output througha single connection port. For example, as shown in FIG. 2A, each of theexcitation light sources may be coupled to an optical fiber, and theoptical fibers may be bundled into a single output connection portconfigured to receive an output fiber optic or liquid light guide. Inother variations, the optics may organize the light emitted by the whitelight provider and/or excitation light provider into two, three, or anysuitable number of optical paths for output. The output of theillumination module may be coupled to one or more of the other modules,such as the surgery-specific module, as described below.

In some variations, the illumination module may be configured such thatsome of the multiple excitation light sources are arranged separatelywithin the module and light from these excitation light sources isdirected into a common module light path at separate points or fromseparate orientations.

In one example, as shown in FIG. 2D, the illumination module may includea first excitation light source 254 a, a second excitation light source254 b, and a white light provider 252, with light from each beingdirected into a common illumination module light path that exits viaport 255 to be connected to a light guide. Light from the excitationlight source 254 a may be directed into the common light path via adichroic mirror 251 a placed in the light path ahead of a blue lightsource 252 c, and light from the excitation light source 254 b may bedirected into the common light path via a mirror 251 e placed behind aset of dichroic mirrors 251 b-d for directing light from the white lightprovider 252 into the common light path. In one embodiment, the firstexcitation light source 254 a may emit a narrow spectrum of light withwavelength about 805 nm (e.g., for excitation of ICG), and the secondexcitation light source 254 b may emit a narrow spectrum of light withwavelength about 675 nm (e.g., for excitation of methylene blue).

In another example, as shown in FIG. 2E, the illumination module mayinclude a first excitation light source 264 a, a second excitation lightsource 264 b, and a white light provider 262, with light from each beingdirected into a common illumination module light path that exits viaport 265 to be connected to a light guide. Light from the excitationlight source 264 a may be directed into the common light path byreflection via dichroic mirror 261 a and dichroic mirror 261 b placed inthe light path ahead of the white light provider 262, while the lightsource 264 b may be directed into the common light path via transmissionthrough dichroic mirror 261 a and reflection via dichroic mirror 261 b.In one embodiment, the first excitation light source 264 a may emit anarrow spectrum of light with wavelength about 805 nm (e.g., forexcitation of ICG), and the second excitation light source 264 b mayemit a narrow spectrum of light with wavelength about 675 nm (e.g., forexcitation of methylene blue).

Optical Image Acquisition Module

As shown in FIGS. 2B-2C, the optical image acquisition module 220 mayinclude camera optics 226 and an image sensor assembly 223. In somevariations, the camera optics 226 may include at least one fluorescenceexcitation light blocking filter 228 and projection optics (e.g., 230 aand 230 b) to project light onto the image sensor assembly 223. As bestshown in FIGS. 2B and 2C, the fluorescence excitation light blockingfilter 228 may be located in the optical path between the object beingimaged and the image sensor assembly 223, in order to substantiallyexclude excitation light from reaching the image sensor assembly. Forinstance, the fluorescence excitation light blocking filter may bephysically integrated as part of the camera optics in the imageacquisition module. In another variation, the fluorescence excitationlight blocking filter may be integrated in a separate optical couplingaccessory that is mounted to the input of the image acquisition moduleand is used to couple any one or more of the surgery-specific modules tothe image acquisition module. In another variation, the fluorescenceexcitation light blocking filter may be integrated with thesurgery-specific modules. However, the fluorescence excitation lightblocking filter may be located in any suitable place in the optical pathbetween the object being imaged and the image sensor assembly.

Blocking Filter and Camera Optics

The fluorescence excitation light blocking filter 228 may blocksubstantially all light in the excitation wavebands (e.g., excitationlight that may be reflected or remitted from the object being imaged)and transmit at least a substantial portion of the white light reflectedby the object and fluorescent light emitted by the fluorophores in theobject. The fluorescence excitation light blocking filter 228 may be amulti-band notch filter to block light in the non-overlapping excitationwavebands. For example, as shown in FIG. 4B, in a system in which theillumination module emits light at excitation wavebands according toBands 1, 2, 3, 4, 5, and 6 described in FIG. 3A, the fluorescenceexcitation light blocking filter may selectively substantially blocklight in Bands 1, 2, 3, 4, 5, and 6 while substantially transmittinglight in all other wavebands. Similarly, in a system in which theillumination module additionally emits light at excitation wavebandsaccording to Bands 1-7 described in FIG. 3A, the fluorescence excitationlight blocking filter may selectively substantially block light in Bands1-7, while substantially transmitting light in wavebands other than oneor more of Bands 1-7. In some variations, the fluorescence excitationlight blocking filter may be characterized by an optical density (OD) ofat least about 4 when blocking the fluorescence excitation wavebands ofthe illumination spectrum. For instance, the fluorescence excitationlight blocking filter may have an OD of 4, 5, or 6, or greater. In somevariations, the fluorescence excitation light blocking filter may becharacterized as having high transmission (e.g., about 90% or greater)in parts of the spectrum other than the excitation wavebands.Furthermore, in some variations, the fluorescence excitation lightblocking filter may be characterized as having steep transition regions(e.g., a transition width less than about 10 nm) between substantiallytransmitted and substantially blocked portions of the light spectrum.However, the fluorescence excitation light blocking filter may have anysuitable OD for blocking fluorescence excitation wavebands, any suitabletransmission rate in the non-excitation waveband portions of thespectrum, and any suitable transition region between substantiallytransmitted and substantially blocked portions of the light spectrum.

For instance, the fluorescence excitation light blocking filter 228 mayinclude at least one substrate (e.g., glass substrate) with one or moredielectric coatings, which may be configured, alone or in combination,to substantially block light in a selected waveband (e.g., by includinga material with a refractive index suitable for preventing transmissionof the selected waveband through the coating). By including multipledielectric blocking coatings, the fluorescence excitation light blockingfilter 228 may substantially block or prevent passage of light in aplurality of selected fluorescence excitation wavebands corresponding tothe filter characteristics of multiple dielectric coatings, whilesubstantially transmitting light in other wavelengths. For example, asshown in FIG. 9, the fluorescence excitation light blocking filter maybe a multi-band notch filter 910 having multiple dielectric coatings 911with alternating high and low refractive indexes on a glass substrate912, which collectively block multiple wavebands of light correspondingto excitation of multiple types of fluorophores. Additionally oralternatively, multiple single-notch blocking filters with differentdielectric coatings may be combined (e.g., placed in series) so as toblock multiple wavelengths.

As shown in the two variations of FIGS. 2B and 2C, the camera optics 226may include projection optics (e.g., 230 a and 230 b) that project lightonto the image sensor assembly 223. More specifically, the projectionoptics may project onto the image plane of the image sensor assemblylight that is transmitted by the fluorescence excitation blocking filter(including reflected white light and emitted fluorescent light). Forexample, the projection optics may include various lenses and/or mirrorsto direct the transmitted light onto the image sensor assembly, and/orany other suitable optical components.

Image Sensor Assembly

The image sensor assembly 223 in the optical image acquisition module220 may include one or more image sensors and various sensor electronics(not shown). In some variations, the image sensor assembly 223 mayinclude solid state image sensors, but in other variations the imagesensor assembly may additionally or alternatively include any suitablenon-solid state image sensors. In some variations, the solid state imagesensors may be at least high definition (HD) or ultra-high definition(4K) in spatial resolution, but in other variations the image sensorsmay have any suitable resolution.

The image sensor assembly 223 may include one or more image sensorsconfigured to detect light at least in the UV, visible and/ornear-infrared I (NIR-I) wavebands (e.g., below about 900 nm). Inparticular, in one variation, the image sensor assembly 223 may includea single solid state image sensor comprising technology such assilicon-based CMOS technology, CCD technology, CID technology, etc. Forexample, the image sensor may be a monochrome image sensor. As anotherexample, as shown in FIG. 2C, the image sensor may be a color imagesensor with an appropriate color filter array (CFA) whose elements aredeposited on the sensor pixels. The CFA may include, for example, aBayer pattern with RGB (red, green, blue), CMYG (cyan, magenta, yellow,green), or WRGB (white, red, green, blue) filters. In another variation,as shown in FIG. 2B, the image sensor assembly may consist of three (orother suitable number) solid state image sensors each including CMOStechnology, CCD technology, CID technology, etc., which may be arrangedon (or in the optical path following) a Philips prism or other spectralsplitting technology.

In some variations, the image sensor assembly 223 may additionally oralternatively include one or more image sensors configured to detectlight at least in the near-infrared II (NIR-II) waveband (e.g., aboveabout 900 nm). Image sensors that detect NIR-II light may be used, forexample, to image tissue at a greater depth beneath the surface oftissue than other image sensors (e.g., sensors that only detect UV,visible, and/or NIR-I light).

In one example, the image sensor assembly 223 may include at least oneindium gallium arsenide (InGaAs) image sensor and/or germanium (Ge)image sensor configured to detect light at least in the NIR-II waveband.As shown in FIG. 10, an InGaAs image sensor or Ge image sensor maydetect light with wavelengths generally between about 650 nm and about1700 nm, with high detection sensitivity for light generally in theNIR-II waveband (e.g., between about 900 nm and 1700 nm). In somevariations, the InGaAs image sensor or Ge image sensor may be used incombination with an image sensor that detects light outside of theNIR-II waveband (e.g., light in the visible and/or NIR-I wavebands) suchthat the image sensor assembly 223 is configured to detect a widerspectrum of light for visible and/or fluorescence imaging.

As another example, the image sensor assembly 223 may include at leastone “black silicon” image sensor (e.g., XQE series of CMOS image sensorsproduced by SiOnyx LLC). As shown in FIG. 10, black silicon imagesensors may detect light with wavelengths generally between about 400 nmand about 1600 nm, with high detection sensitivity for light generallyincluding visible and NIR light between about 600 nm and about 1200 nm,which is further into the NIR-II waveband than what is detected withsome other silicon image sensors. In some variations, a single blacksilicon image sensor may be used for both reflected visible light colorimaging and for fluorescence imaging in the NIR-I and/or NIR-IIwavebands. In these variations, the image sensor signals correspondingto reflected visible light images may be extracted and formed into colorimages through spatial means or temporal image processing methods. Forinstance, the black silicon image sensor may have a CFA coupled to ordeposited on its sensor pixels, where color images (reflected visiblelight images) may be formed by spatial image processing techniques(e.g., demosaicing and spatial interpolation between pixels of the samecolor, as further described below). As another example, in variations inwhich a single black silicon image sensor is used for both reflectedvisible light color imaging and for fluorescence imaging in the NIR-Iand/or NIR-II wavebands, the black silicon image sensor may lack a CFAbut provide for formation of color images through temporally-based imageprocessing techniques (e.g., synchronized pulsing and image sensorreadout schemes, as further described below).

As another example, the system may include at least one upconverter thattransforms incident light in at least the NIR-II waveband into visiblelight. For example, as shown in FIG. 11A, an upconverter 1110 mayinclude an NIR photodetector 1111 and an organic light-emitting diode1116 (OLED) coupled to the photodetector 1111, where the photodetector1111 and OLED 1116 are configured to up-convert incident NIR light 1113(e.g., NIR-I and/or NIR-II light) into converted visible light 1114. Forexample, the photodetector 1111 and OLED 1116 may be configured in amanner similar to that described in U.S. Patent Pub. No. 2012/0286296 orin U.S. Patent Pub. No. 2014/0217284, the contents of which areincorporated in their entirety by this reference. However, the systemmay additionally or alternatively include other suitable NIR-to-visibleupconverters. The upconverter 1110 may, in some variations, be furtherconfigured to transmit incident visible light 1117. The convertedvisible light 1114 and/or transmitted visible light 1117 maysubsequently be received and detected by one or more sensors in theimage sensor assembly. In some variations, the system may includemultiple upconverters in an array. For example, as shown in FIG. 11B, aNIR-to-visible upconverter array 1120 may include a plurality ofupconverters 1110 that receive incident NIR light 1113, convert theincident NIR light 1113 into visible light, and emit converted visiblelight 1114 toward at least one image sensor 1115. The image sensor 1115may, for example, be a silicon-based CMOS or CCD sensor. In somevariations, the upconverter array 1120 may further transmit incidentvisible light such that the image sensor 1115 detects both thetransmitted visible light including information for the white lightimage and the converted visible light for the fluorescence imagesignals.

At least one image sensor that detects light in at least the UV,visible, and/or NIR-I wavebands may be combined with at least one imagesensor that detects light in at least the NIR-II waveband. For example,the image sensor assembly may include one or more image sensors thatdetect fluorescence emission in any of Bands 1-7 shown in FIG. 3A, inany combination. In some of these variations in which the image sensorassembly is configured for broad spectrum imaging in the UV, visible,NIR-I, and/or NIR-II spectrums, the optics in the optical imageacquisition module 220, any accessory modules 130 in the image path,and/or the surgery-specific modules 120 may be coated to substantiallytransmit light in these wavebands (with the exception of wavelengthsblocked by the fluorescence excitation light blocking filter, etc.).Furthermore, in some of these variations, the optical design of thesemodules may be corrected to provide images that display minimal opticalaberration across the transmitted spectrum.

In some variations, the image sensor assembly may include multiple imagesensors arranged on (or in the optical path following) a Philips prismor other spectral splitting technology. The prism or other beamsplittersmay receive incident light (which may include UV, visible, NIR-I, and/orNIR-II light, etc.) and spectrally distribute the incident light ontothe multiple image sensors such that each image sensor receives asubspectrum of light transmitted by the fluorescence excitation lightblocking filter. These sensors may be arranged in several differentconfigurations including, but not limited to, a two-sensor,three-sensor, four-sensor, or five-sensor configurations, such that eachimage sensor receives a subspectrum of the light transmitted by thefluorescence excitation blocking filter.

In one variation, the image assembly may include a two-sensor prismconfiguration including a beam splitter that divides incident light intotwo subspectrums of light. For example, as shown in FIG. 12A, thetwo-sensor prism configuration may include a beam splitter 1200 a thatreceives and spectrally divides incident light 1210 transmitted by thefluorescence excitation light blocking filter into a first branch towarda first sensor 1220 and a second branch toward a second sensor 1230. Forinstance, first sensor 1220 may be configured to detect NIR-I and/orNIR-II light for the fluorescence image and second sensor 1230 may be acolor image sensor with a CFA or a monochrome image sensor configured todetect visible light for the white light image. However, the two-sensorprism configuration may include sensors configured to detect anysuitable subspectrums of light that are formed by the beam splitter.

In another variation, the image assembly may include a three-sensorprism configuration including a beam splitter that divides incidentlight into three subspectrums of light. For example, as shown in FIG.12B, the three-sensor prism configuration may include a beam splitter1200 b that receives and spectrally divides incident light 1210transmitted by the fluorescence excitation light blocking filter into afirst branch toward a first sensor 1220, a second branch toward a secondsensor 1230, and a third branch toward a third sensor 1240. Forinstance, the first sensor 1220 may be configured to detect blue light,the second sensor 1230 may be configured to detect green light, and thethird sensor 1240 may be configured to detect red light, where thesignals for detected blue, green, and red light may be combined for afull white light or color image. As another example, the first sensor1220 may be configured to receive NIR-I light for a first fluorescenceimage, the second sensor 1230 may be a color sensor with a CFAconfigured to receive visible light for the white light image, and thirdsensor 1240 may be configured to receive NIR-II light for a secondfluorescence image. However, the three-sensor prism configuration mayinclude sensors configured to detect any suitable subspectrums of lightthat are formed by the beam splitter.

In another variation, the image assembly may include a four-sensor prismconfiguration including a beam splitter that divides incident light intofour subspectrums of light. For example, as shown in FIG. 12C, thefour-sensor prism configuration may include a beam splitter 1200 c thatreceives and spectrally divides incident light 1210 transmitted by thefluorescence excitation light blocking filter into a first branch towarda first sensor 1220, a second branch toward a second sensor 1230, athird branch toward a third sensor 1240, and a fourth branch toward afourth sensor 1250. For instance, the first sensor 1220 may beconfigured to detect blue light, the second sensor 1230 may beconfigured to detect green light, the third sensor 1240 may beconfigured to detect red light, and the fourth sensor 1250 may beconfigured to detect NIR-I or NIR-II light. In this example, the signalsfor detected blue, green, and red light may be combined for a full whitelight or color image, while the signals for the detected NIR-I or NIR-IIlight may be for a fluorescence image. However, the four-sensor prismconfiguration may include sensors configured to detect any suitablesubspectrums of light that are formed by the beam splitter.

In another variation, the image assembly may include a five-sensor prismconfiguration including a beam splitter that divides incident light intofive subspectrums of light. For example, as shown in FIG. 12D, thefive-sensor prism configuration may include a beam splitter 1200 d thatreceives and spectrally divides incident light 1210 transmitted by thefluorescence excitation light blocking filter into a first branch towarda first sensor 1220, a second branch toward a second sensor 1230, athird branch toward a third sensor 1240, a fourth branch toward a fourthsensor 1250, and a fifth branch toward a fifth sensor 1260. Forinstance, the first sensor 1220 may be configured to detect blue light,the second sensor 1230 may be configured to detect green light, thethird sensor 1240 may be configured to detect red light, the fourthsensor 1250 may be configured to detect NIR-I light, and the fifthsensor 1260 may be configured to detect NIR-II light. In this example,the signals for detected blue, green, and red light may be combined fora full white light or color image, while the signals for the detectedNIR-I and NIR-II light may be for fluorescence images. However, thefour-sensor prism configuration may include sensors configured to detectany suitable subspectrums of light that are formed by the beam splitter.

Sensor Electronics

Sensor electronics may include sensor readout control electronics thatadjust the operation of the sensor. The image sensor assembly mayadditionally or alternatively include image signal managementelectronics (e.g., amplifier, digitizer, memory, serializer, etc.) toprepare the electronic image signal for transmission to the controllerand/or image processor module. However, in some variations, theseelectronics may be located outside of the image sensor assembly itself,and instead in any suitable location (e.g., as part of the controller,etc.).

Other Modules

As shown in FIG. 1, the system may include a controller module, a 3Dcontroller module, a processor module, a 3D processor module, and/or apost-processor/data manager module 110 c. The controller module maycommunicate with, control, and synchronize the operation of theillumination module and the optical image acquisition module, and/or anyother components that involve coordination for capturing images. Thecontroller module may include an internal clock to enable control of thevarious elements and help establish correct timing of illumination andsensor shutters.

The processor module may receive the electronic image signal from theimage acquisition module and process (e.g., in real-time or nearreal-time) the signal to create images and/or other data, such as foroutput to display and/or recording. In some variations, the controllermodule and/or processor module may be embodied on any computer orcomputing means such as, for example, a tablet, laptop, desktop,networked computer, dedicated standalone microprocessor, etc. Forinstance, the controller module and/or processor module may include oneor more central processing units (CPU). In some variations, thecontroller module and processor module may be integrated as a combinedcontroller and processor module 110 b as shown in FIG. 1.

In some variations in which the system includes a stereoscopicsurgery-specific module (e.g., stereoscopic videoscope or otherstereoscopic surgical device as described further below), the system mayinclude a 3D controller module and/or 3D processor module for roboticsapplications (in this case the regular controller processor may not beutilized) which subsequently outputs a 3D image data signal to theappropriate 3D compatible accessory modules (displays, recorders, etc.).In some variations, the 3D controller module and 3D image processingmodule may be integrated as a combined 3D controller and 3D processormodule 110 a as shown in FIG. 1.

The post-processor/data manager module 110 c may receive the images fromthe processor (or 3D processor) and perform additional processing steps,such as overlaying of white light images and fluorescence images, orotherwise modifying images, as further described below with respect tothe operation of the fluorescence imaging system. The postprocessor/data manager module 110 c may additionally or alternativelymanage the output of image data generated (e.g., with respect to thedata modules 140). Although the post-processor/data manager module 110 cmay be embodied in a physical unit separate from the controller moduleand/or image processor (or 3D controller module and/or 3D imageprocessor) as pictured in FIG. 1, in other variations, thepost-processing module 110 c may be integrated with any of the othermodules. Furthermore, the post-processor/data manager module 110 c maybe divided into separate modules (e.g., one post-processor module andone data manager module).

Surgery-Specific Modules

As shown in FIG. 1, the fluorescence imaging system may include one ormore surgery-specific modules 120. Each surgery-specific module may beprimarily designated for a particular kind or category of surgicalapplication, and may be interchangeable with other surgery-specificmodules and/or selectable such that the fluorescence imaging system is aplatform configurable by an operator (e.g., clinician) for a particularkind of surgical procedure. In many instances, the surgery-specificmodules may be largely opto-mechanical in nature, but need not be. Thesurgery-specific modules may interconnect indirectly (e.g., via lightguide 130 a) or directly with one or more of the modules to direct thewhite light and excitation light to the object and collect reflectedwhite light and emitted fluorescent light from the object. In somevariations, an accessory module (e.g., light guide) may transmit thereflected white light and emitted fluorescent light to the imageacquisition module. In other variations, the surgery-specific module maydirectly transmit the reflected white light and fluorescence emission tothe image acquisition module without a separate accessory module.

One variation of the surgery-specific module may include a surgicalmicroscope 120 d with the appropriate magnification and working distancefor microsurgical applications. The surgical microscope may beconfigured for electronic image capture with the optical imageacquisition module, and/or may also provide a direct viewing binocularoption.

Another variation of the surgery-specific module may include alaparoscope/endoscope 120 b, such as for minimally invasive orendoscopic surgeries.

Another variation of the surgery-specific module may include an openfield illumination/imaging module 120 c, such as for laparotomy/openfield surgery. In some variations, the open field illumination/imagingmodule may be handheld and/or supported by a positioning/supporting armor robotic arm 130 c. In these variations, the handheld aspects, and/orthe positioning arm or robotic arm may be provided in an accessorymodule (or integrated as part of the surgery-specific module).

Another variation of the surgery-specific module may include astereoscopic videoscope 120 a, such as for robotics applications. Forexample, a stereoscopic device may interconnect two image acquisitionmodules to a stereoscopic laparoscope, either with or without a separateoptical coupler (which may be an accessory module or integrated in thesurgery-specific module). In some variations, the stereoscopic devicemay include a dedicated stereoscopic camera.

Another variation of the surgery-specific module may include ascintigraphy module 120 e. Further variations include modules designatedor specially-designed for other suitable kinds of surgical applications.

In some variations, the surgery-specific modules and optical imageacquisition module may be integrated. For instance, the camera opticsand sensor assembly of the optical image acquisition module may beintegrated with the surgery-specific module (e.g., laparoscope/endoscopemodule, surgical microscope module, wide field illumination/imagingmodule, stereoscopic laparoscope module for robotic surgeryapplications, etc.). In these variations, at least some of the sameremaining operative modules (and the one or more accessory modules tointerconnect these with the factory-integrated image acquisition andsurgery-specific modules) may be utilized.

Accessory Modules

One or more accessory modules 130 may be interconnected with theoperative and/or surgery-specific modules and provide additionalfunctions. One variation of an accessory module includes an opticalconnection or light guide 130 a (e.g., fiber optic, liquid light guide,etc.) for delivering light from the illumination module to thesurgery-specific module. Another variation of an accessory moduleincludes an optical connection or light guide (e.g., fiber optic, liquidlight guide, etc.) for delivering light captured by the surgery-specificmodule to the imaging acquisition module. Another variation of anaccessory module includes a coupler 130 b that couples one or more ofthe surgery-specific modules (e.g., 120 a, 120 b, 120 c, and/or 120 d,etc.) to the optical image acquisition module 220. The coupler may, forexample, mount to the surgery-specific module and the optical imageacquisition module to indirectly join these two modules together. Thecoupler may or may not include an optical connection or light guide fordelivering light to and/or from the surgery-specific module. Yet othervariations of accessory modules may provide mechanical support (e.g.,support arm 130 c) or enclosure for one or more of the modules, aidingin the interconnection/adaptation of other modules, and/or othersuitable functions not provided by the operative modules orsurgery-specific modules.

Data Modules

The system may include one or more data modules 140 that receive imagedata. As shown in FIG. 1, one variation of a data module includes avideo display 140 a or other monitor (e.g., computer monitor, touchscreen, etc.) that enables display of substantially real-time and/orrecorded image and data to a clinician, patient, or other user. Anothervariation of a data module includes a recorder 140 b (e.g., hard disk,flash memory, other tangible non-transitory computer readable medium,etc.) or other data storage device that can store images and/or otherdata. Another variation of a data module includes a printer 140 c forcreating hard copies of images and/or other data for furthervisualization, archiving, record-keeping, or other purposes. Yet anothervariation of a data module includes a picture archiving andcommunication system 140 d (PACS) which may, for example, store data instandard Digital Imaging and Communications in Medicine (DICOM) formator any other suitable format. Other variations of data modules includesystems for communicating and/or storing image data in any suitablemanner.

Operation of the Fluorescence Imaging System

The operation of an intraoperative fluorescence imaging system, such asa system configured as an interconnected set of operative modules, oneor more surgery-specific modules, and one or more accessory modules asdescribed above, will now be described. In some variations, the imagingsystem may be a multi-mode system in that it can operate in any one of anon-fluorescence mode, fluorescence mode, and a combined fluorescenceand non-fluorescence mode. Each of these modes is described below. Inother variations, the imaging system may be a single mode system thatoperates only in the fluorescence mode, which may be similar to thefluorescence mode in the multi-mode system operation described below.

In a non-fluorescence mode of operation, the fluorescence imaging systemmay provide real time full color visible (white) light image data fordisplay on a video monitor and/or for recording. In this mode, theillumination module provides broad visible spectrum light output (e.g.,via solid state components such as laser diodes, filtered LEDs, filterednon-solid state light sources, etc., or a combination of these) whichmay be coupled to and transmitted by the surgery-specific module andprojected onto the surface to be illuminated. The broad visible spectrumlight reflected from the illuminated surface may be captured by thesurgery-specific module and transmitted to the image acquisition modulethat transduces the image data. The transduced electronic image signalmay be subsequently transmitted to the image processor that processesand outputs for display and/or recording in real time, with negligiblelatency. The displayed and/or recorded reflected white light image datamay have a high color fidelity, such that it is a highly accurate colordepiction of the surface that is reflecting the light. These images maybe displayed and/or recorded in full color and at high definition (HD)or ultra-high definition (UHD or 4K) resolution (or other suitableresolution). This full color, white light imaging mode may be optionalfor some surgeries, such as those in which the surgeon has a direct lineof site to the surgical area and/or does not require an anatomicalcontext in which to assess the fluorescence image data.

In a fluorescence mode of operation, the fluorescence imaging systemprovides real time fluorescence emission image data for display on avideo monitor and/or for recording. In this mode, the illuminationmodule provides fluorescence excitation light output (e.g., via solidstate components such as laser diodes, filtered LEDs, filtered non-solidstate light sources, etc. or a combination of these) which may becoupled to and transmitted by the surgery-specific module and projectedonto the surface to be illuminated. The fluorescence emission emanatingfrom the excited fluorophores within the illuminated area may becaptured by the surgery-specific module and transmitted to the imageacquisition module that transduces the image data. The transducedelectronic image signal may be subsequently transmitted to the imageprocessor that processes and outputs for display and/or recording inreal time, with negligible latency. The displayed and/or recordedfluorescence emission image data may be monochrome (e.g., black andwhite or grayscale) or pseudo-colored (e.g., via a color map based onintensity or some other signal parameter) and may be displayed and/orrecorded in a monochrome or pseudo-colored fashion at high definition(HD) or ultra-high definition (UHD or 4K) resolution (or other suitableresolution). This fluorescence emission imaging mode may be a standalonemode for surgeries in which the surgeon has a direct line of sight tothe surgical area and/or does not require an anatomical context in whichto assess the fluorescence image data.

In a combination non-fluorescence and fluorescence mode of operation,the fluorescence imaging system simultaneously provides the options of(a) real time full color visible (white) light image data, (b) real timefluorescence emission image data, and (c) a combination of real timefull color visible (white) light image data and real time fluorescenceemission image data for display on a video monitor and/or for recording.In this combination mode, the illumination module operatessimultaneously in two illumination modes to provide both broad visiblespectrum light output and fluorescence excitation light output (e.g.,via solid state components such as laser diodes, filtered LEDs, filterednon-solid state light sources, etc. or a combination of these). Thisillumination may be transmitted by the surgery-specific module andprojected onto the surface to be illuminated. The visible light outputand the fluorescence light output are pulsed so that different wavebandsare illuminating the area to be imaged at different times. The pulsingscheme may be such that the broad visible spectrum light andfluorescence excitation light are both pulsed or that only one of theillumination modes (either the broad visible spectrum light or thefluorescence excitation light) is pulsed. Alternatively, some portion ofeither the broad visible spectrum light and/or fluorescence excitationlight may be pulsed.

As a result of the pulsed illumination modes, the illumination of thearea to be imaged by broad visible spectrum light and fluorescenceexcitation light may be composed of any of four kinds of illumination:(1) where the light output is pulsed for both illumination modes suchthat the illumination modes are partially or completely separated intime; (2) where the light output for one illumination mode is continuousand the other mode is pulsed; (3) where a wavelength portion of thelight output for one illumination mode is continuous and the other modeis pulsed; and (4) where the light output is continuous for bothillumination modes. The surgery-specific module captures the broadvisible spectrum light reflected from the illuminated surface and thefluorescence emission emanating from the fluorophores within theilluminated area, and transmits this reflected light and fluorescenceemission to the image acquisition module that transduces the image data.The transduced electronic image signal is subsequently transmitted tothe image processor, which separates the image signal associated withthe reflected broad visible spectrum light from the image signalassociated with the fluorescence emission. The processing scheme in theimage processor is synchronized and matched to the pulsing scheme in theillumination module (e.g., via the controller) to enable this separationof the image signals. The rate of pulsing and image processing may besuch that the processed image signals are output for display and/orrecording in real time, with negligible latency.

For example, as shown in FIGS. 17A and 17B, in some variations (e.g., inwhich the image acquisition module includes a single image sensor thatmay receive visible light and/or fluorescence emission), the pulsedlight output for both the visible light (“RGB”) and fluorescence(“Laser”) illumination modes may be synchronized with the acquisition ofvisible light (“Exp (VIS)”) and fluorescence emission (“Exp (FL)”),respectively, by the image sensor. As shown in FIG. 17B, in somevariations the system may further compensate for background illumination(e.g., room lighting) in substantially similar wavelengths as thefluorescence emission. In these variations, visible light andfluorescence emission imaging may be performed while reducing the riskof confounding the fluorescence emission with background lighting. Forinstance, the system may include at least one image sensor configured todetect background light corresponding to the same or similar wavelengthsas the fluorescence emission, such that the signal for this detectedbackground light can be subtracted from signals provided by any sensorthat detects the fluorescence emission. Alternatively, as shown in FIG.17C, in some variations (e.g., in which the image acquisition moduleincludes one or more fluorescence emission image sensors in addition toa reflected visible light image sensor), continuous light output forboth the visible light and fluorescence illumination modes maycorrespond with the separate, continuous acquisition of visible light(“Exp (VIS)”), NIR-I fluorescence emission (“Exp (NIR I FL)”), and/orNIR-II fluorescence emission (“Exp (NIR II FL)”) by the image sensors.Additional examples of such pulsing and image processing schemes havebeen described in U.S. Pat. No. 9,173,554, filed on Mar. 18, 2009 andtitled “IMAGING SYSTEM FOR COMBINED FULL-COLOR REFLECTANCE ANDNEAR-INFRARED IMAGING,” the contents of which are incorporated in theirentirety by this reference. However, other suitable pulsing and imageprocessing schemes may be used.

The image data from the broad visible spectrum light may be processed byany suitable color image processing methods according to the nature ofthe image acquisition module. In variations in which the imageacquisition module includes a color camera with a single solid stateimage sensor and a color filter array deposited on the sensor surface,the image processing method may include de-mosaicing the color imagesignal, followed by amplification, A/D conversion, and/or storage incolor image memory. The typical (but not the only) signal format aftersuch processing is luminance/chrominance (Y_(c), c_(r) c_(b)) format. Invariations in which the image acquisition module includes a color camerawith three solid state image sensors mounted on a Philips (RGB) prism(or other beam splitting element), the image processing method mayinclude receiving a direct readout of the red, green, and blue colorimage from the camera, followed by amplification, A/D conversion, and/orstorage in color image memory. The typical (but not the only) signalformat after such processing is luminance/chrominance (Y_(c), c_(r)c_(b)) format.

The processed image data may be output in a multi-window (e.g., tiled,matrix) display and/or recorded in high definition (HD) or ultra-highdefinition (UHD or 4K) resolution (or any suitable resolution), withnegligible latency. The color image data and fluorescence image data maybe simultaneously output in separate channels for display and/orrecording. Similar to the white light images in thenon-fluorescence-only mode, the displayed and/or recorded reflectedwhite light image data may have a high color fidelity, such that it is ahighly accurate color depiction of the surface that is reflecting thelight. Similar to the fluorescence images in the fluorescence-only mode,the displayed and/or recorded fluorescence emission image data may bemonochrome (e.g., black and white or grayscale) or pseudo-colored (e.g.,via a color map based on intensity or some other signal parameter) andmay be displayed and/or recorded in a monochrome or pseudo-coloredfashion. Additionally or alternatively, the white light image data andthe fluorescence image data may be overlaid or otherwise combined. Forexample, the fluorescence emission image data may be used to modify thechrominance (c_(r), c_(b)) in the white light image data such thatpixels with higher fluorescence signal intensity are increasinglysaturated by a non-naturally occurring color (e.g., green in biologicalsystems).

Method for Fluorescence Imaging

In some variations, the method for fluorescence imaging an object mayinclude emitting white light, emitting excitation light in a pluralityof excitation wavebands for causing the object to emit fluorescentlight, directing the white light and excitation light to the objectand/or collecting reflected white light and emitted fluorescent lightfrom the object, blocking substantially all light in the excitationwavebands and transmitting at least a substantial portion of thereflected white light and/or the fluorescent light, and receiving thetransmitted reflected white light and fluorescent light on an imagesensor assembly. In some variations, the white light and excitationlight may be directed to the object and/or reflected white light andemitted fluorescent light may be collected from the object by acomponent (e.g., an interchangeable, surgery-specific component). Insome variations, the reflected white light and fluorescent lightreceived at the image sensor assembly may be temporally and/or spatiallymultiplexed. In some variation, excitation light is emitted in aplurality of non-overlapping excitation wavebands for causing the objectto emit fluorescent light.

In some variations, the method may include receiving image signals fromthe image sensor assembly, and processing the received image signals togenerate images from the received image signals. In some variations, themethod may include controlling the white light provider and/orexcitation light provider to operate in a non-fluorescence mode, afluorescence mode, or a combined non-fluorescence and fluorescence mode.In these variations, the processing steps may include separating imagesignals from the image sensor assembly into a first set of image signalsassociated with the reflected white light and a second set of imagesignals associated with the fluorescent light. The processing steps mayfurther include generating a white light image based on the first set ofimage signals and a fluorescence emission image based on the second setof image signals.

Other aspects of the method include performing any of the various stepsand functions described above with respect to the operation of thefluorescence imaging system with a configurable platform, and/or thefunctions of various components therein.

Multiplexed Fluorescence Imaging System

As shown in FIG. 5, in some variations, a multiplexed fluorescenceimaging system 500 for imaging an object includes: a light sourceassembly 510 including a white light provider 512 that emits white lightand an excitation light provider 514 that emits excitation light in aplurality of excitation wavebands for causing the object 502 to emitfluorescent light; a camera 520 with at least one image sensor 540 thatreceives reflected white light and emitted fluorescent light from theobject; an optical assembly 530 located in the optical path between theobject and the image sensor, wherein the optical assembly 530 includes afirst optics region that projects the reflected white light as a whitelight image onto the image sensor and a second optics region thatreduces the image size of the fluorescent light, spectrally separatesthe fluorescent light, and projects the separated fluorescent light asfluorescence images onto different portions of the image sensor; and animage processor 550 that electronically magnifies the fluorescenceimages. As a result, the white light image and multiple fluorescentlight images may be simultaneously projected onto an image plane (forone or more image sensors) in a single camera in a spatially andtemporally multiplexed manner. As a result, the multiplexed fluorescenceimaging system 500 can, simultaneously and in real time, acquirefluorescence emission images at multiple wavelengths within the visibleand NIR spectrum, as well as acquire full color reflected light (whitelight) images. Furthermore, this functionality may be achieved with theuse of only a single camera, thereby reducing bulk of the overall systemand enabling the system to be used in a greater variety of surgicalapplications. In some variations, the excitation light provider 514emits excitation light in a plurality of non-overlapping excitationwavebands for causing the object 502 to emit fluorescent light

Although the components of the system are primarily described below asgrouped in particular assemblies, in some variations, the variouscomponents may be organized and grouped in any suitable manner (that is,the various components described herein may be combined and arranged inassemblies and subassemblies different from those described herein).Furthermore, in some variations, the components may be combined in asingle physical system (e.g., an imaging system for use in a clinicalsetting). In other variations, some or all of the components (e.g., theimage processor) may be located separate from the other components, suchas on a computer system at an off-site location that is remote from aclinical site or otherwise not embodied in the same physical unit as theother components.

Light Source Assembly

As shown in FIG. 5, in some variations, the multiplexed fluorescenceimaging system 500 may include a light source assembly 510 including awhite light provider 512 and an excitation light provider 514.

The white light provider 512 emits white light for illumination of theobject to be imaged. In some variations, the white light providerincludes one or more solid state emitters such as LEDs and/or laserdiodes. For example, the white light provider may include blue, green,and red LEDS or laser diodes that in combination generate visible(white) light illumination. In some variations, these light sources arecentered around the same wavelengths (e.g., ˜460 nm, ˜530 nm, and ˜635nm) around which the camera (described further below) is centered. Forexample, in variations in which the camera includes a single chip,single color image sensor having an RGB color filter array deposited onits pixels, the blue, green, and red light sources may be centeredaround the same wavelengths around which the RGB color filter array iscentered. As another example, in variations in which the camera is athree-chip, three-sensor (RGB) color camera system, the blue, green, andred light sources may be centered around the same wavelengths aroundwhich the blue, green, and red image sensors are centered.

The excitation light provider 514 emits fluorescence excitation light ina plurality of excitation wavebands that are non-overlapping. One ormore of the excitation wavebands may be selected such that it fallsoutside of the visible white light spectrum used for imaging (betweenabout 450 nm and about 650 nm), so that a fluorescence excitation lightblocking filter (described further below) substantially blocking anyremitted/reflected excitation wavelengths in the imaging path does notalso substantially block white light reflected from the object, andtherefore will not substantially interfere with the generation of awhite light image. In some variations, at least some of the excitationwavebands may at least partially overlap with the visible spectrum,which may result in some compromise of reflected white light ultimatelyimaged (since some of the reflected white light may be blockedsimultaneously with any remitted/reflected excitation light whosewavelength overlaps with the reflected white light), but such excitationwavebands may nevertheless be suitable. For example, in variations inwhich the excitation light provider emits excitation light in a wavebandcentered at about 470 nm, a fluorescence excitation light blockingfilter that substantially blocks any remitted/reflected excitation lightin that waveband may also at least partially block cyan wavelengths,which is only a segment of the entire white light spectrum.

In some variations, the excitation light provider 514 may emit light inexcitation wavebands centered at (i) about 670 nm to excite Cy5, Cy5.5,Methylene Blue, porphysomes, etc.; (ii) about 770 nm to excite NIRfluorophores such IRDye800, etc. and/or NIR-II fluorophores such asIR-PEG, etc.; (iii) about 805 nm to excite ICG (Indocyanine Green) oranalogues such as IfCG, etc. and/or NIR-II fluorophores such as IR1061or CH1100, etc.; (iv) about 405 nm to excite tissue auto-fluorescence,etc.; and/or (v) about 470 nm to excite Fluorescein, Vitamin B2, etc. Inone exemplary embodiment, the excitation light provider emits light inexcitation wavebands (i), (ii), and (iii) described above. In anotherexemplary embodiment, the excitation light provider emits light inexcitation wavebands (i), (ii), (iii), and one or both of wavebands (iv)and (v). However, the excitation light provider may emit light in anynumber and any combination of wavebands (i), (ii), (iii), (iv), and (v).The excitation light provider may additionally or alternatively emitlight centered around any suitable wavelength.

In some variations, the excitation light provider 514 includes solidstate light sources, such as laser diodes or LEDs. Solid state elementsmay have a number of advantages in the multiplexed fluorescence imagingsystem described herein. In particular, solid state elements can berapidly switched on and off, and their duty cycle (time on vs. time off)can be altered electronically. Additionally, laser diodes emit lightalong relatively narrow spectral lines which can be effectively andprecisely blocked with one or more commensurately narrow excitationlight blocking filters in the imaging path when Stokes shifts are short,such that the one or more excitation light blocking filters do notsubstantially interfere with the collection of other wavelengths whenimaging. Finally, solid state light sources provide various otherpractical advantages including lifetime, cost, energy efficiency, easeof adjusting color preferences, etc. However, the excitation lightprovider may additionally or alternatively include non-solid state lightsources in any suitable combination.

In some variations, the light source assembly 510 may be configured as aseries of white light and excitation light emitters whose collimatedoutputs are folded into a combined optical path by one or more dichroicmirrors and/or other suitable optical components. The excitation lightsources (e.g., laser diodes, etc.) may be fiber-coupled so that thelight output from the distal end of the optical fibers can be bundled orotherwise easily positioned for collimation and folding into a singleoptical path. This arrangement may enable a relatively compactconfiguration that contributes to a more compact fluorescence imagingsystem, and which may be more easily cooled for thermal managementpurposes (e.g., by using a single heat spreader plate). However theoutputs of the white light and excitation light emitters may beorganized and transmitted out of the light source assembly in anysuitable manner.

Optical Assembly and Camera

As shown in FIG. 5, the multiplexed fluorescence imaging system mayinclude an optical assembly 530 and a camera system 520 with at leastone image sensor assembly 540. The optical assembly 530 may transmit, inan illumination optical path, the white light and the excitation lightfrom the light source assembly to the object being imaged. The opticalassembly 530 may also receive, in an imaging optical path, reflectedvisible (white) light and emitted fluorescent light in the correspondingwavebands for the fluorophores in the object that are excited by thelight source assembly. In some variations, the optical assembly 530 maymanipulate the reflected white light and/or fluorescence light asdescribed further below, and output the light to the camera system.After receiving the white light and fluorescent light, the camera systemmay transduce the received light into electrical image signals for theimage processor (described below) to process.

The optical assembly 530 may take various form factors depending on thesurgical application. For example, the optical assembly may include alens assembly for wide field (e.g., open surgery) illumination andimaging. As another example, the optical assembly may include a surgicalmicroscope for illuminating and imaging a microscopic field of view. Asanother example, the optical assembly may include an endoscope forilluminating and imaging a surface interior to the body through a smallsurgical opening or via a natural orifice/lumen. In some variations, theoptical assembly may be interchangeable, similar to one or more of thesurgery-specific modules described above in the fluorescence systemhaving a configurable platform. Due to the size and/or weight of thelight source assembly, the light from the light source assembly may begenerally transmitted to the optical assembly by a light guide (e.g.,optical fiber, liquid light guide, etc.), but the light from the sourceassembly may be transmitted to and from the optical assembly in anysuitable manner.

In some variations, the optical assembly 530 and the camera system 520may be separate components. For example, the optical assembly may bepart of a surgical microscope with a removable camera. As anotherexample, the optical assembly may be part of a rigid laparoscope with acamera mounted proximally (e.g., camera mounted on the eyepiece, etc.).In other variations, the optical assembly 530 may be integrated with thecamera system 520. For example, the optical assembly may be integratedwith a wide field camera system for use in open surgery/laparotomy,where the optical assembly and camera system may be mounted on a supportarm, be hand held, or be positionable in any suitable manner. As anotherexample, the optical assembly may be integrated in a video endoscope inwhich the camera is mounted at the distal end of the scope.

As shown in the exemplary variations depicted in FIGS. 6A, 6B, 7, 8, and16A-16C, the combination of the optical assembly and camera, whetherseparate or integrated, may include various optical components locatedin the optical path between the object and one or more image sensors inthe camera. In particular, the optical assembly may include an opticsregion that projects the reflected white light as a white light imageonto the image sensor in the camera, and an optics region that reducesthe image size of the fluorescence light, spectrally separates thefluorescent light, and projects the separated fluorescence light asfluorescence images onto different portions of the image sensor in thecamera. As a result, the white light image and multiple fluorescentlight images may be simultaneously projected onto an image plane (withone or more image sensors) in a single camera, in a spatially andtemporally multiplexed manner. The optical assembly may includeadditional optical components such as beam splitters, mirrors, etc. thatalso manipulate white light and/or emitted fluorescent light before thelight is projected onto the image sensor.

As shown in FIGS. 6A-6B, in some variations, the optical assembly mayinclude a field lens 602 and/or other input optics that capture lighttraveling in the imaging path from the object toward the camera. Thislight may include, for example, reflected white light illumination,reflected or remitted excitation light illumination, emitted fluorescentlight originating from excited fluorophores in the object, and/or otherlight in other wavebands that are traveling in the imaging path.

As shown in FIGS. 6A-6B, in some variations, the optical assembly mayinclude a fluorescence excitation light blocking filter 610 thatsubstantially exclude excitation light from reaching the image sensor.The fluorescence excitation light blocking filter may be a multi-bandnotch filter that blocks substantially all fluorescence excitation lightproduced by the light source assembly (which may be reflected orremitted from the object), but passes at least a substantial portion ofvisible (white) light for color imaging and at least a substantialportion of the fluorescence emission bands of fluorophores excited bythe fluorescence excitation light. The filter 610 may be located in theoptical imaging path between the fluorophores in the object and the oneor more image sensors in the camera system, such that only the reflectedwhite light and the emitted fluorescent light will be projected onto theone or more image sensors. In some variations, the filter may be locatedin a portion of the optical path in which the light rays have a minimalcone angle. In some variations, the filter may be a multi-layerinterference filter, though in other variations the filter may have anysuitable construction.

The optical assembly may include additional optics regions forperforming various beam shaping functions described below. In somevariations, the optical assembly may include a dichroic or other kind ofbeam splitter that may separate the light transmitted by thefluorescence excitation light blocking filter into white light andfluorescent light components. In particular, the beam splitter maydivide the optical path into at least two legs or branches: one branchfor reflected visible (white) light that is transmitted by thefluorescence excitation light blocking filter, and at least one branchfor emitted fluorescence light that is transmitted by the fluorescenceexcitation light blocking filter. However, the beam splitter 612 mayfurther divide (or not further divide) the fluorescent light transmittedby the fluorescence excitation light blocking filter into multiplefluorescent optical paths. In one variation, as shown in FIGS. 6A and6B, a dichroic splitter 612 may transmit visible light 604 and reflectfluorescence light 606, thereby diverting fluorescence light to adifferent path (e.g., one that is offset from the optical axis of theimage sensor).

As shown in FIGS. 6A-6B, in some variations, the optical assembly mayinclude demagnification optics 614 that reduce the image size of theemitted fluorescent light. The demagnification optics may include, forexample, one or more lens systems that reduce the size of thefluorescence images. Once reduced, the multiple fluorescence images cansubsequently be detected simultaneously with the white light image bythe same image sensor assembly, as further described below. In anexemplary embodiment, the demagnification optics reduce the imagedimensions of the emitted fluorescent light by an approximate factor of2, thereby causing the dimensions of each of the fluorescence images tobe about one-half the corresponding dimensions of the white light image(i.e., such that each of the fluorescence images has an image area aboutone-fourth the image area of the white light image). In othervariations, the demagnification optics may reduce the image dimensionsof the emitted fluorescent light by any suitable factor, which may ormay not depend on the number of excitation/emission wavebands used bythe system. In some instances, the demagnified fluorescent light may beredirected or otherwise shaped by other optical components such asmirror 616 that redirects the fluorescent light toward beam splitter618. However, in variations in which multiple fluorophores havingnon-overlapping emitted light wavebands are excited by a commonexcitation wavelength, then the beam splitter 618 may divide thefluorescence emission into light paths corresponding to the distinctemission wavebands.

As shown in FIGS. 6A-6B, in some variations, the optical assembly mayinclude one or more additional beam splitters 618 that further separatethe fluorescence emission optical path following the demagnificationoptics. The beam splitters 618 may include one or more dichroic mirrors,prisms, other suitable beam splitters, or any suitable combination orassembly thereof. The beam splitter 618 may be designed and/or selectedto spectrally separate the fluorescence emission generated by theexcitation wavelengths (e.g., at ˜670 nm, ˜770 nm and ˜805 nm, etc.)into separate, demagnified fluorescent image paths. For instance, abeam-splitting prism may include multiple portions (e.g., components)that have dimensions and/or include a material chosen (based on factorssuch as refractive index) in order to equalize the optical path lengthfor all split beam paths. For example, a vertical beam-splitting prismmay be used to divide the fluorescence emission into multiple lightpaths to be offset vertically. As shown in FIGS. 13A and 13B, a verticalbeam-splitting prism 1310 may include multiple portions (e.g., 1311,1312, 1313, and 1314) that spectrally separate incident light 1315 intoat least two vertically offset light paths 1316 and 1317. Additionallyor alternatively, a horizontal beam-splitting prism 1410 may be used todivide the fluorescence emission into multiple light paths to be offsethorizontally. As shown in FIGS. 14A and 14B, a horizontal beam-splittingprism may include multiple portions (e.g., 1411, 1412, 1413, and 1414)that spectrally separate incident light 1415 into at least twohorizontally offset light paths 1416 and 1417.

In an exemplary embodiment, the beam splitter 618 divides thefluorescence emission into four light paths corresponding to fourexcitation wavelengths that generated the fluorescence emission. In somevariations, this may be achieved with a beam-splitting prism assembly1510 including a combination of prism beam splitters. For example, asshown in FIG. 15, a beam-splitting prism assembly 1510 may include onehorizontal beam-splitting prism 1511 (similar to horizontalbeam-splitting prism 1410) in combination with two verticalbeam-splitting prisms 1511 a and 1511 b (similar to verticalbeam-splitting prism 1310). In particular, the horizontal beam splittingprism 1511 may split an incident fluorescent light branch into twohorizontally offset fluorescent light branches, each of which isreceived by a respective vertical beam-splitting prism 1512 a or 1512 b.Each of the two vertical beam-splitting prisms 1512 a and 1512 b maysubsequently split its received fluorescent light branch into twovertically offset fluorescent light branches, thereby resulting in fourfluorescence light branches. These four fluorescent light branches maythen be directed onto four quadrants of the image plane at an imagesensor.

In some variations, the optical assembly may include an alignmentcomponent system containing at least one dichroic element or otheralignment component that realigns the multiple fluorescence emissionoptical paths and the visible light optical path prior to the imagesensor(s), such that separate fluorescence images are projected ontodifferent portions of the image plane at the sensor. As shown in FIGS.6A and 6B, in variations in which the fluorescence light was previouslydiverted away from the visible light, such an alignment component systemmay fold fluorescence emission optical paths back into the visible lightoptical path. In particular, the alignment component system may includemirror 620 a and dichroic mirror 620 b that reflect the multiplefluorescence branches into the same optical path as the white lightbranch 604. As a result, the alignment components may cause the whitelight and the spectrally separated fluorescent light to follow the sameoptical path toward the image sensor(s) 640, and the optical assembly asa whole may project the full-sized white light image and the demagnifiedfluorescence images simultaneously onto the image sensor(s) in thecamera. In an exemplary embodiment (e.g., where the fluorescence imagesare demagnified by an approximate factor of two), the alignmentcomponent system may cause the four demagnified fluorescence images tobe projected onto four (4) quadrants of the image plane at the sensor(s)640. However, the alignment component system may cause the fluorescenceimages to be projected on any suitable portions of the image plane.

As shown in FIGS. 6A-6B, in some variations, the optical assembly mayinclude projection optics 622 for the visible (white) light that projectthe visible (white) light image into the image plane at the one or moreimage sensors in the camera sensor and/or for the demagnified,spectrally separated fluorescence light. The projection optics 622 mayinclude, for example, any suitable combination of lenses, mirrors,filters, or other optical components suitable for projecting the lightonto the one or more image sensors.

In another variation as shown in FIGS. 16A-16C, the optical assembly maybe similar to that depicted in FIGS. 6A-6B except described below. Inparticular, incident light 1610 (e.g., excitation light, visible light,emitted fluorescence light, etc.) passes through field lens 1611 orother input optics, and then through fluorescence excitation lightblocking filter 1612 that prevents passage of fluorescence excitationlight. In contrast to the dichroic splitter 612 shown in FIGS. 6A and6B, dichroic splitter 1613 a transmits fluorescence light 1620 andreflects visible light 1619, thereby diverting the visible light to adifferent path (e.g., one that is offset from the optical axis of theimage sensor). The fluorescence light branch 1620 transmitted by thedichroic splitter 1613 a continues into demagnification optics 1615 andbeam splitter assembly 1616 which spectrally divides the demagnifiedfluorescence light into four branches 1623 a, 1623 b, 1623 c, and 1623d. The visible light branch 1619 reflected by the dichroic splitter 1613a may be diverted by components such as mirror 1614 a to maintainsubstantially equal optical path length for the visible and fluorescencelight paths. An alignment component system (e.g., mirror 1614 b anddichroic mirror 1613 b) may fold the visible light branch 1619 into thesame optical path as the four fluorescence branches such that thevisible light and fluorescence light pass through projection optics1617. Projection optics 1617 projects the visible (white) light imageonto the center of the image plane at image sensor 1618 and projects thefour fluorescence images onto four quadrants of the image plane at theimage sensor. However, the alignment component system may cause thefluorescence images to be projected on any suitable portions of theimage plane.

Although the above components are primarily described as arranged in aparticular order in the optical path, the optical assembly componentsmay be arranged such that the various beam splitting, demagnification,and alignment steps (or subset thereof) may occur in any suitable mannerand combination. For example, in some variations, the beam splitter (#1)may further split the emitted fluorescence light into multiple branches(e.g., two, three, four, etc.) before the demagnification optics. Forexample, the beam splitter may divide the emitted fluorescence lightsuch that each branch of fluorescent light corresponds to a respectiveexcitation waveband (e.g., about 670 nm, about 770 nm, about 805 nm,etc.) that caused the fluorophores in the object to emit the fluorescentlight. In these variations, the optical assembly may include multiplesets of demagnification optics, each of which may reduce the image sizeof a respective fluorescent optical branch. In these variations, theoptical assembly may omit one or more beam splitters (#2) since nofurther division of the fluorescent light may be necessary followingdemagnification.

The camera of the fluorescence system may include an image sensorassembly for transducing the full color visible (white) light opticalimage and de-magnified fluorescence emission images projected onto thefour quadrants of the sensor/sensor assembly. The image sensor assemblymay have high definition or ultra-high definition spatial resolution(e.g., 4K or higher resolution). In some variations, as shown in FIG.6A, the image sensor assembly may include a single sensor 640 a (e.g.,with a color filter array). In some variations, as shown in FIG. 6B, theimage sensor assembly may include a three-sensor assembly 640 b, whichmay be coupled to a Philips prism or other spectral splittingtechnology. In some variations, the camera may include an image sensorassembly similar to that described above in the fluorescence imagingsystem with a configurable platform. However, the camera may include anysuitable kind of image sensor assembly.

In some variations, some or all of the optics regions for performing thevarious beam functions described above (e.g., projecting the reflectedwhite light as a white light image onto the image sensor, reducing theimage size of the fluorescence light, spectrally separating thefluorescent light, projecting the separated fluorescence light asfluorescence images onto different portions of the image sensor, etc.)may be combined in one or more prisms, in addition to or instead ofseparate components. The one or more prisms may be made of any suitablekind of optical glass or other suitable kind of material that transmitslight.

FIG. 7 illustrates one variation in which an optical assembly 700includes a prism 720. Field lens 702 and fluorescence excitation lightblocking filter 710 may be similar to field lens 602 and filter 610described above with respect to FIGS. 6A-6B. Prism 720 may have facetsor other structures that split the incoming light into at least two legsor branches: one branch 704 for reflected visible (white) light, asecond branch 706 a for fluorescence light of one emission waveband, anda third branch 706 b for fluorescence light of another emissionwaveband. In particular, region A spectrally splits fluorescent light ofWaveband A into branch 706 a, region B spectrally splits fluorescentlight of Waveband B into branch 706 b, and white light of Waveband Cpasses into region C in branch 704.

Prism 720 may further include regions D and E, which define beam-shapingprism faces including demagnification optics (e.g., 714 a, 714 b, 714 c,and 714 d, etc.). In some variations, each concave or other suitabledemagnifying prism face may demagnify by a factor of about the squareroot of 2, such that in order to reduce the dimensions of a fluorescenceimage by an overall factor of 2, the fluorescence image may interactwith two beam-shaping prism faces (e.g., branch 706 a is shaped by prismfaces 714 a and 714 b, while branch 706 b is shaped by prism faces 714 cand 714 d). However, the fluorescent light may interact with anysuitable number of beam-shaping prism faces to achieve any suitablelevel of demagnification. Generally speaking, these demagnificationoptics may result in demagnified fluorescence emission images, similarto demagnification optics 614 described above with respect to FIGS.6A-6B.

Prism 720 may further include regions F and G, which may fold themultiple fluorescence emission optical paths back into the visible lightoptical path prior to the image sensor(s) 740, similar to mirrors 620 aand 620 b described above with respect to FIGS. 6A-6B. The opticalassembly may further include projection optics 722 and one or more imagesensors 740, which may be similar to projection optics 622 and imagesensor(s) 640 a and/or 640 b described above with respect to FIGS.6A-6B.

The various regions A-G of prism 720 may have differing indices ofrefraction to compensate for differing travel distances for the whitelight branch 704 and the fluorescent light branches 706 a and 706 b. Inother words, the differing indices of refraction may substantiallyequalize the travel time/optical path length for the white light branch704 and the fluorescent light branches 706 a and 706 b. In particular,regions D and E may have a lower index of refraction than region C, suchthat light traveling through regions D and E will reach projectionoptics 722 and image sensor(s) 740 at the same time as light travelingthrough region C. However, the regions A-G of prism 720 may have anysuitable combination of materials with varying index of refraction suchthat the white light branch 704 and fluorescent light branches 706 a and706 b have about equal travel times. Furthermore, in other variations,the prism 720 may have additional or fewer regions corresponding todifferent numbers of excitation/emission wavebands of fluorescent lightthat will be separated, demagnified, and projected onto the image sensor(e.g., two additional regions similar to regions D and E, for shapingfour separate paths of fluorescent light for four excitation/emissionwavebands). Additionally, in some variations, prism 720 may comprisemultiple prisms in combination.

FIG. 8 illustrates another variation of an optical assembly 800 which issimilar to optical assembly 700 depicted in FIG. 7 and described above,with at least the following differences where noted below. Field lens802, filter 810, projection optics 822, and image sensor(s) 840 may besimilar to field lens 702, filter 710, projection optics 722, and imagesensor(s) 740 described above with respect to FIG. 7, respectively.Furthermore, prism 820 may include regions A, B, C, F, and G similar tothe corresponding regions in prism 720. However, prism 820 may omitregions D and E in prism 720 (and omit demagnification optics regions714 a, 714 b, 714 c, and 714 d), and instead prism 820 may include otherseparate demagnification optics (e.g., 814 a, 814 b, 814 c, and 814 d)to demagnify the image sizes of fluorescent light in branches 806 a and806 b. Similar to the prism 720, region C of prism 820 may have a higherindex of refraction to equalize the travel time/optical path length forthe white light branch 804 and fluorescent light branches 806 a and 80b.

Yet other variations of the optical assembly may include any suitablecombination of the variations shown in FIGS. 6A, 6B, 7, 8, and/or16A-16B, and/or may include additional optics to separate thefluorescent light into more than the three branches described in theabove examples.

Controller and Image Processor

As shown in FIG. 5, the multiplexed fluorescence imaging system 500 mayinclude a controller 560 and an image processor 550. The controller 560may control the light source assembly 510 such that either the whitelight provider 512, or fluorescence excitation light provider 514, orboth, are strobed at a high frequency (e.g., 60 Hz or greater),preferably in synchronous operation with the image acquisition by thecamera. The white light and the excitation light may be pulsed at thesame or different frequencies. The camera may have an appropriatelymatching sensor read-out frequency and acquire either separate whitelight and fluorescence emission images, and/or a known combination ofvisible light and fluorescence emission images which can be separated byfurther image processing (e.g., by comparing image frames with strobedillumination/excitation light on and off). The high speed strobing ofthe illumination and read-out of the camera sensors may enable thefluorescence emission and full color white light image data to besimultaneously displayed in real time.

The image processor 550 may receive the transduced image signals fromthe camera and process them into white light and fluorescence images. Inparticular, the image processor may electronically magnify thefluorescence images to restore their image size to about their originalsize before demagnification. The electronic magnification may cause theimage size of the fluorescence images to be about the same size as thewhite light image. In some variations, the image processor may spatiallyco-register the magnified fluorescence images with the white lightimage.

Display and Other Data Components

As shown in FIG. 5, in some variations, the multiplexed fluorescenceimaging system may include one or more data components 570 such as adisplay, recorder, or other data storage device, printer, and/or PACSsimilar to the data modules described above with respect to thefluorescence imaging system with configurable platform. The multiplexedfluorescence imaging system may additionally or alternatively includeany other suitable systems for communicating and/or storing image data.

In some variations, the white light images and/or fluorescence imagesmay be displayed on a high definition or ultra-high definition display(e.g., on a monitor having 4K or higher spatial resolution). Thefluorescence images may be displayed in one or more of multiple manners.The manner in which the fluorescence images are displayed may beselected by an operator in a user interface. In one variation, thefluorescence images can be individually displayed as monochrome images.In another variation, the chroma of each of the fluorescence images canbe mapped to different contrasting color for each fluorescence emission,where the mapped color is chosen to be one that is not likely to occurnaturally in the body (e.g., green, purple, etc.). The fluorescenceimages can then be individually or collectively combined with the fullcolor, visible (white) light image for display. In another variation,the intensity of the fluorescence signal in a fluorescence image can benormalized by scaling the brightness (luma) of each of the fluorescenceimages with the co-registered reflected red light image signal (i.e.,the red portion of the full visible (white) light image), and thendisplayed with a color map selected to emphasize specific ranges offluorescence intensity.

Similarly, in some variations, one or more of the other data components(e.g., data storage module or recorder, printer, PACS, etc.) cancommunicate and/or store the white light images and the fluorescenceimages as they appear in any of the above-described manners.

Method for Fluorescence Imaging an Object

A method for fluorescence imaging an object may include emitting whitelight, emitting excitation light in a plurality of excitation wavebands,causing the object to emit fluorescent light, receiving reflected whitelight and emitted fluorescent light from the object on an at least oneimage sensor, and feeding at least part of the reflected light throughan optical assembly located in an optical path between the object andthe image sensor. The method may include projecting reflected whitelight as a white light image onto the image sensor. The method mayinclude reducing the image size of the fluorescent light, spectrallyseparating the fluorescent light, and projecting the separatedfluorescent light as fluorescence images onto different portions of theimage sensor. In some variations, the method includes electronicallymagnifying (e.g., with an image processor) at least some of thefluorescence images. In some embodiments, excitation light is emitted ina plurality of non-overlapping excitation wavebands.

A kit may include any part of the systems described herein (includingcomponents of variations of the fluorescence imaging system with aconfigurable platform, components of variations of the multiplexedfluorescence imaging system, or combinations of components thereof) anda fluorescence imaging agent such as, for example, a fluorescence dyesuch as ICG or any suitable fluorescence imaging agent. The kit mayinclude instructions for use of at least some of its components (e.g.,for using the fluorescence imaging agent, operating the fluorescenceimaging system, maintaining the fluorescence imaging system, etc). Inyet further aspects, there is provided a fluorescence imaging agent suchas, for example, a fluorescence dye, for use in the systems and methodsdescribed herein.

While the present disclosure has been illustrated and described inconnection with various embodiments shown and described in detail, it isnot intended to be limited to the details shown, since variousmodifications and structural changes may be made without departing inany way from the scope of the present disclosure. Various modificationsof form, arrangement of components, steps, details and order ofoperations of the embodiments illustrated, as well as other embodimentsof the disclosure may be made without departing in any way from thescope of the present disclosure, and will be apparent to a person ofskill in the art upon reference to this description. It is thereforecontemplated that the appended claims will cover such modifications andembodiments as they fall within the true scope of the disclosure. Forthe purpose of clarity and a concise description, features are describedherein as part of the same or separate embodiments, however, it will beappreciated that the scope of the disclosure includes embodiments havingcombinations of all or some of the features described. For the terms“for example” and “such as,” and grammatical equivalences thereof, thephrase “and without limitation” is understood to follow unlessexplicitly stated otherwise. As used herein, the singular forms “a”,“an”, and “the” include plural referents unless the context clearlydictates otherwise.

What is claimed is:
 1. A fluorescence imaging system for imaging anobject, comprising: a white light provider that emits white light; anexcitation light provider that emits excitation light in a plurality ofexcitation wavebands for causing the object to emit fluorescent light,wherein the excitation light provider is distinct form the white lightprovider and can emit excitation light simultaneously with the whitelight provider emitting white light; a component that directs the whitelight and excitation light to the object and collects reflected whitelight and emitted fluorescent light from the object; a filter thatblocks light in the excitation wavebands and transmits at least aportion of the reflected white light and fluorescent light; an imagesensor assembly that receives the transmitted reflected white light andthe fluorescent light; and an optical assembly located in an opticalpath between the object and the image sensor assembly, the opticalassembly comprising: a first optics region that projects the transmittedreflected white light as a white light image onto an image plane of theimage sensor assembly, and a second optics region for reducing an imagesize for the transmitted fluorescent light, spectrally separating thetransmitted fluorescent light, and projecting the separated fluorescentlight as fluorescence images onto different portions of an image sensorof the image sensor assembly.
 2. The system of claim 1, wherein at leastone of the excitation wavebands is centered at about 405 nm, about470-480 nm, about 660 nm, about 760-780 nm, about 805 nm, or about750-810 nm.
 3. The system of claim 1, wherein the excitation lightprovider comprises at least three excitation light sources.
 4. Thesystem of claim 1, wherein at least a portion of the excitation lightprovider is coupled to an optical filter that narrows the spectrum oflight emitted from the excitation light provider.
 5. The system of claim1, wherein the filter has an optical density of at least
 4. 6. Thesystem of claim 1, wherein the filter transmits at least 90% of thereflected white light and the fluorescent light.
 7. The system of claim1, wherein the image sensor assembly comprises a single image sensor. 8.The system of claim 1, wherein the image sensor assembly comprises aplurality of image sensors.
 9. The system of claim 1, wherein thecomponent is an interchangeable component.
 10. The system of claim 1,further comprising an image processor and at least one controller thatcontrols the system to selectively operate in a non-fluorescence mode, afluorescence mode, or a combined non-fluorescence and fluorescence mode,wherein: in the non-fluorescence mode, the controller causes the whitelight provider to emit white light and the image processor generates awhite light image based on image signals associated with the reflectedwhite light from the object, in the fluorescence mode, the controllercauses the excitation light provider to emit excitation light and theimage processor generates a fluorescence emission image based on imagesignals associated with the fluorescent light from the object, and inthe combined non-fluorescence and fluorescence mode, the controllercauses at least a portion of the white light or at least a portion ofthe excitation light to be pulsed.
 11. The system of claim 1, whereinthe reflected white light and the fluorescent light received at theimage sensor are temporally multiplexed, spatially multiplexed, or bothtemporally multiplexed and spatially multiplexed.
 12. A method forfluorescence imaging of an object, comprising: emitting white light by awhite light emitter; emitting excitation light in a plurality ofexcitation wavebands from an excitation light provider that is distinctfrom the white light emitter for causing the object to emit fluorescentlight; directing the white light and excitation light to the object;collecting reflected white light and emitted fluorescent light from theobject; blocking light in the excitation wavebands and transmitting atleast a portion of the reflected white light and fluorescent light; anddirecting the transmitted reflected white light and fluorescent lightthrough an optical assembly located in an optical path between theobject and an image sensor assembly, wherein: a first optics region ofthe optical assembly projects the transmitted reflected white light as awhite light image onto an image plane of the image sensor assembly, anda second optics region reduces an image size for the fluorescent light,spectrally separates the fluorescent light, and projects the separatedfluorescent light as fluorescence images onto different portions of animage sensor of the image sensor assembly.
 13. The method of claim 12,wherein at least one of the excitation wavebands is centered at about405 nm, about 470-480 nm, about 660 nm, about 760-780 nm, about 805 nm,or about 750-810 nm.
 14. The method of claim 12, comprising emittingexcitation light in a plurality of excitation wavebands from at leastthree excitation light sources.
 15. The method of claim 12, furthercomprising narrowing the spectrum of emitted excitation light.
 16. Themethod of claim 12, comprising blocking light in the excitationwavebands using a filter has an optical density of at least
 4. 17. Themethod of claim 12, comprising blocking at least 90% of the reflectedwhite light and the fluorescent light.
 18. The method of claim 12,wherein the image sensor assembly comprises a single image sensor. 19.The method of claim 12, wherein the image sensor assembly comprises aplurality of image sensors.
 20. The method of claim 12, comprisingdirecting the white light and excitation light to the object using aninterchangeable component.
 21. The method of claim 12, furthercomprising: emitting white light and generating a white light imagebased on image signals associated with the reflected white light fromthe object in a white light mode; emitting excitation light andgenerating a fluorescence emission image based on image signalsassociated with the fluorescent light from the object in a fluorescencemode; and pulsing at least a portion of the white light or at least aportion of the excitation light in a combined non-fluorescence andfluorescence mode.
 22. The method of claim 12, further comprisingtemporally multiplexing the reflected white light and the fluorescentlight, spatially multiplexing the reflected white light and thefluorescent light, or both temporally multiplexing and spatiallymultiplexing the reflected white light and the fluorescent light. 23.The system of claim 1, wherein the optical assembly comprises thefilter.
 24. The system of claim 1, wherein the optical assemblycomprises a beam splitter that separates the transmitted reflected whitelight and fluorescent light into a first branch of reflected white lightand a second branch of fluorescent light.
 25. The system of claim 1,wherein the second optics region comprises a beam splitter thatspectrally separates the fluorescent light.
 26. The system of claim 25,wherein the optical assembly comprises at least one demagnificationoptic for reducing the image size for the transmitted fluorescent lightand the beam splitter is downstream of at least one demagnificationoptic.
 27. The system of claim 1, further comprising an image processorconfigured to electronically magnify the fluorescence images.
 28. Themethod of claim 12, wherein the optical assembly comprises a beamsplitter that separates the transmitted reflected white light andfluorescent light into a first branch of reflected white light and asecond branch of fluorescent light.
 29. The method of claim 12, whereinthe second optics region comprises a beam splitter that spectrallyseparates the fluorescent light.
 30. The method of claim 12, furthercomprising electronically magnifying the fluorescence images.
 31. Thesystem of claim 1, wherein the first and second optics regions projectthe transmitted reflected white light and the transmitted fluorescencelight, respectively, onto the same image sensor.
 32. The method of claim12, wherein the first and second optics regions project the transmittedreflected white light and the transmitted fluorescence light,respectively, onto the same image sensor.